ext3-2.4-0.9.6
Patch against linux-2.4.8 is at
http://www.uow.edu.au/~andrewm/linux/ext3/
The only changes here are merging up to 2.4.8 and the bigendian fix.
linux-2.4.8-ac1 currently has ext3-0.9.3 which has no known crash-worthy bugs, but is old. I'm about to send Alan a diff which takes -ac up to 0.9.6. The changes between 0.9.3 and 0.9.6 may be summarised as:
- Simplify the handling of synchronous operations (O_SYNC, fsync(), chattr +S, etc).
- Fix a couple of places where we're not syncing writes when we should.
- Implement batching of synchronous operations: when multiple threads want to perform synchronous writes we allow the threads to block together
and all their writes happen in the same transaction. Speeds things up
muchly.
- Implement support for external journal devices. This is experimental at this stage. It works fine, but the operational interfaces will change. At present the external journal device is not "mounted" when we're using it and it really should be.
- ext3 has for a long time had developer code which allows the target device to be turned read-only at the disk device driver level a certain number of jiffies after the fs was mounted. This is to allow scripted testing of crash recovery. This facility has been extended to support two devices; one for the filesystem and one for the external journal device.
- Accelerate an O(N^2) algorithm in log_do_checkpoint().
- Accelerate an O(N^2) algorithm in journal_commit_transaction().
- Rate-limit some error messages which can come out when we're hopelessly out of memory.
- Honour __GFP_WAIT in journal_try_to_free_buffers(). The fs is supposed to perform synchronous writeout on the second pass of page_launder() and we weren't doing that - we were starting all IO async. The net effect of this change is to decrease throughout with dbench by 10-20%, but system CPU time goes from 60% to 30%. It's the right thing to do...
Quoted 6 lines · Andrew Morton
Patch against linux-2.4.8 is at
http://www.uow.edu.au/~andrewm/linux/ext3/The only changes here are merging up to 2.4.8 and the bigendian fix.
Gack. I think about when you wrote this, I managed to crash again. I was running 2.4.8-pre8 + fsync_dev -> fsync_no_super + first fix. It was at transaction.c:1184, but the logs didn't make it to disk. On a related note, what does ext3 do to the disk when this happens, I think I need to point the yaboot author at it since it couldn't load a kernel (which was fun, let me tell you.. :))
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 5 lines · Tom Rini
Quoted 2 lines · Andrew Morton
Patch against linux-2.4.8 is at
Quoted 8 lines
Quoted 4 lines
http://www.uow.edu.au/~andrewm/linux/ext3/The only changes here are merging up to 2.4.8 and the bigendian fix.
Gack. I think about when you wrote this, I managed to crash again. I was running 2.4.8-pre8 +
fsync_dev->fsync_no_super+ first fix. It was at transaction.c:1184, but the logs didn't make it to disk.
I'd assumed that this was related to the endianness fix. You're sure you were running with that in place? If you can capture a buffer trace that'd be great.
Quoted 4 lines
On a
related note, what does ext3 do to the disk when this happens, I think I need to point the yaboot author at it since it couldn't load a kernel (which was fun, let me tell you.. :))
ext3 is designed to nicely crash the machine if it thinks something may be wrong with the fs - it's very defensive of your data.
If yaboot is open firmware's native ext2 capability then presumably it refuses to read an ext3 partition which needs recovery. ext3 is designed to not be compatible with ext2 when it's in the needs-recovery state.
Probably the simplest way to avoid this is to make the boot partition ext2.
-
Quoted 11 lines · Andrew Morton
Quoted 10 lines · Tom Rini
Quoted 6 lines · Andrew Morton
Patch against linux-2.4.8 is at
http://www.uow.edu.au/~andrewm/linux/ext3/The only changes here are merging up to 2.4.8 and the bigendian fix.
Quoted 7 lines
Quoted 3 lines
Gack. I think about when you wrote this, I managed to crash again. I was running 2.4.8-pre8 +
fsync_dev->fsync_no_super+ first fix. It was at transaction.c:1184, but the logs didn't make it to disk.I'd assumed that this was related to the endianness fix. You're sure you were running with that in place? If you can capture a buffer trace that'd be great.
I'm sure I had the fix in. I re-ran the original test I had a few times and it was good. I'll try and capture the buffer trace if it happens again, but last time I'm guessing it happened on my root fs, so the log couldn't goto disk.
Quoted 5 lines
Quoted 4 lines
On a
related note, what does ext3 do to the disk when this happens, I think I need to point the yaboot author at it since it couldn't load a kernel (which was fun, let me tell you.. :))
Quoted 7 lines
ext3 is designed to nicely crash the machine if it thinks something may be wrong with the fs - it's very defensive of your data.
If yaboot is open firmware's native ext2 capability then presumably it refuses to read an ext3 partition which needs recovery. ext3 is designed to not be compatible with ext2 when it's in the needs-recovery state.
It's the linux bootloader that OF runs. Is there any 'safe' way to read data off of an unclean ext3 partition? I'm thinking grub might run into this problem too..
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 10 lines · Tom Rini
...
Quoted 3 lines
I'd assumed that this was related to the endianness fix. You're sure you were running with that in place? If you can capture a buffer trace that'd be great.
I'm sure I had the fix in. I re-ran the original test I had a few times and it was good. I'll try and capture the buffer trace if it happens again, but last time I'm guessing it happened on my root fs, so the log couldn't goto disk.
OK, thanks.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
On a
related note, what does ext3 do to the disk when this happens, I think I need to point the yaboot author at it since it couldn't load a kernel (which was fun, let me tell you.. :))ext3 is designed to nicely crash the machine if it thinks something may be wrong with the fs - it's very defensive of your data.
Quoted 8 lines
Quoted 4 lines
If yaboot is open firmware's native ext2 capability then presumably it refuses to read an ext3 partition which needs recovery. ext3 is designed to not be compatible with ext2 when it's in the needs-recovery state.
It's the linux bootloader that OF runs. Is there any 'safe' way to read data off of an unclean ext3 partition? I'm thinking grub might run into this problem too..
Well, LILO works OK with an unclean ext3 FS because it goes straight to the underlying blocks. If both grub and OF parse the superblock compatibility bits then they could fail in this manner.
I *think* that at present an unrecovered ext3 filesystem is "incomaptible" with ext2. If, however we were to make it "read-only compatible" then ext2-aware loaders would still be able to read the fs and boot from it. But this stuff makes my head hurt - let's see what Andreas and Stephen have to say.
-
Quoted 5 lines · Andrew
I *think* that at present an unrecovered ext3 filesystem is "incomaptible" with ext2. If, however we were to make it "read-only compatible" then ext2-aware loaders would still be able to read the fs and boot from it. But this stuff makes my head hurt - let's see what Andreas and Stephen have to say.
I advocated changing the compat flag to be RO_COMPAT at one time as well. Technically, an unrecovered ext3 filesystem is as "compatible" as an ext2 filesystem that was not fscked before mount. We ro mount unchecked root filesystems all the time, so there shouldn't be a _huge_ issue for ro mounting unrecovered ext3 filesystems.
Stephen and Ted disagree, because with the ext3 journal it is possible to have a large number of pending changes in the journal at the time of a crash. The Linux VFS doesn't easily allow flushing all of the cached inodes between recovery and remount-rw, so this may cause filesystem corruption if the in-kernel inode data does not match the on-disk inode data after recovery.
The only time this becomes an issue is with the root filesystem, generally.
The "solution" for the problem at hand would probably be to make GRUB, et. al., recognize the INCOMPAT_RECOVER flag, and still read the kernel/initrd images from disk. They will generally be static, so the contents of the journal will not affect them, and can be ignored.
Cheers, Andreas
Signature · 3 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- DogbertQuoted 5 lines · Andrew Morton
- ext3 has for a long time had developer code which allows the target device to be turned read-only at the disk device driver level a certain number of jiffies after the fs was mounted. This is to allow scripted testing of crash recovery. This facility has been extended to support two devices; one for the filesystem and one for the external journal device.
Would this facility also be able to deal with parts of a device becoming read-only unexpectedly? Some of the disks I have in RAIDs have the nice habit of disabling write access when overheating. That's an interesting failure scenario in a RAID system.
Ralf
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Tom Rini
Quoted 12 lines
Quoted 4 lines
On a
related note, what does ext3 do to the disk when this happens, I think I need to point the yaboot author at it since it couldn't load a kernel (which was fun, let me tell you.. :))ext3 is designed to nicely crash the machine if it thinks something may be wrong with the fs - it's very defensive of your data.
If yaboot is open firmware's native ext2 capability then presumably it refuses to read an ext3 partition which needs recovery. ext3 is designed to not be compatible with ext2 when it's in the needs-recovery state.
Quoted 7 lines
Quoted 3 lines
It's the linux bootloader that OF runs. Is there any 'safe' way to read data off of an unclean ext3 partition? I'm thinking grub might run into this problem too..
Well, LILO works OK with an unclean ext3 FS because it goes straight to the underlying blocks. If both grub and OF parse the superblock compatibility bits then they could fail in this manner.
Both GRUB and yaboot can read directly from the fs. It's possible to boot a kernel right out of OF from an HFS partition (which I had to do to get the box up again). It might be worth documenting this someplace.
Quoted 3 lines
I *think* that at present an unrecovered ext3 filesystem is "incomaptible" with ext2. If, however we were to make it "read-only compatible" then ext2-aware loaders would still be able to read the fs and boot from it.
That'd be nice. There's lots of PPC boxes which don't have a seperate /boot. But ext2 should be able to read a clean ext3, yes? I never get a chance to check thing... :)
Signature · 2 lines
Tom Rini (TR1265) http://gate.crashing.org/~trini/
Quoted 9 lines · Ralf Baechle
Quoted 5 lines · Andrew Morton
- ext3 has for a long time had developer code which allows the target device to be turned read-only at the disk device driver level a certain number of jiffies after the fs was mounted. This is to allow scripted testing of crash recovery. This facility has been extended to support two devices; one for the filesystem and one for the external journal device.
Quoted 4 lines
Would this facility also be able to deal with parts of a device becoming read-only unexpectedly? Some of the disks I have in RAIDs have the nice habit of disabling write access when overheating. That's an interesting failure scenario in a RAID system.
Well, that facility is purely for development purposes. The obvious way of testing recovery is to hit the reset button at strategic times, which rather sucks. So what the above IDE driver trick does is adds a new mount option `ro-after=3000'. When this is provided, a kernel timer fires 30 seconds after mount and the IDE driver starts silently ignoring writes to the underlying device. It also provides a special ioctl() which blocks the caller until the timer has fired.
So we have scripts which do:
1: mount fs, set to go read-only in 30 seconds
2: start some filesystem activity
3: Block on the timer
4: wake up, kill off the filesystem activity
5: unmount the fs
6: mount the fs (this will run recovery)
7: unmount the fs
8: fsck it
9: repeat with a different read-only interval.
I also have a hacked-on version of dbench which writes known-but-variable info into the files, so we can check that the contents of whatever files survived the "crash" are correct.
This setup has allowed me to run crash+recovery many thousands of times with varying workloads - I'm pretty confident about recovery because of this. The one thing it doesn't cover is the effects of disk write caching.
As for the RAID problem: if the filesystem has magically turned read-only then all you need to do is to unmount it (often hard to do, if it's in use), then make it writable and then mount it or run fsck against it. ext3 will perform recovery and all should be peachy, until next time...
-
Quoted 8 lines · Tom Rini
Quoted 3 lines · Andrew Morton
Well, LILO works OK with an unclean ext3 FS because it goes straight to the underlying blocks. If both grub and OF parse the superblock compatibility bits then they could fail in this manner.
Both GRUB and yaboot can read directly from the fs. It's possible to boot a kernel right out of OF from an HFS partition (which I had to do to get the box up again). It might be worth documenting this someplace.
Same goes for SILO too. Not sure if SILO even works with ext3 right now (uses libext2fs, so I assume so).
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- ...on that fantastic voyage... -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Hi,
Quoted 8 lines · Ralf Baechle
Quoted 5 lines · Andrew Morton
- ext3 has for a long time had developer code which allows the target device to be turned read-only at the disk device driver level a certain number of jiffies after the fs was mounted. This is to allow scripted testing of crash recovery. This facility has been extended to support two devices; one for the filesystem and one for the external journal device.
Quoted 4 lines
Would this facility also be able to deal with parts of a device becoming read-only unexpectedly? Some of the disks I have in RAIDs have the nice habit of disabling write access when overheating. That's an interesting failure scenario in a RAID system.
No, that particular part of the ext3 patch is only there for testing --- it forces a single device readonly to simulate a crash. It adds no ability to deal cleanly with a device unexpectedly going readonly on its own.
Cheers,
Stephen