Proposal and plan for ext2/3 future development work
Given the recent discussion on LKML two weeks ago, it is clear that many people feel they have a stake in the future development plans of the ext2/ext3 filesystem, as it one of the most popular and commonly used filesystems, particular amongst the kernel development community. For this reason, the stakes are higher than it would be for other filesystems. The concerns that were expressed can be summarized in the following points:
* Stability. There is a concern that while we are adding new features, bugs might cause developers to lose work. This is particularly a concern given that 2.6 is a "stable" kernel series, but traditionally ext2/3 developers have been very careful even during development series since kernel developers tend to get cranky when all of their filesystems get trashed.
* Compatibility confusion. While the ext2/3 superblock does have a very flexible and powerful system for indicating forwards and backwards compatibility, the possibility of user confusion has caused concern by some, to the point where there has been one proposal to deliberately break forwards compatibility in order to remove possible confusion about backwards compatibility. This seems to be going too far, although we do need to warn against kernel and distribution-level code from blindly upgrading users' filesystems and removing the ability for those filesystems to be mounted on older systems without an explicit user approval step, preferably with tools that allow for easy upgrading and downgrading.
* Code complexity. There is a concern that unless the code is properly factored, that it may become difficult to read due to a lot of conditionals to support older filesystem formats.
Unfortunately, these various concerns were sometimes mixed together in the discussion two months ago, and so it was hard to make progress. Linus's concern seems to have been primarily the first point, with perhaps a minor consideration of the 3rd. Others dwelled very heavily on the second point.
To address these issues, after discussing the matter amongst ourselves, the ext2/3 developers would like to propose the following path forward.
1) The creation of a new filesystem codebase in the 2.6 kernel tree in /usr/src/linux/fs/ext4 that will initially register itself as the "ext3dev" filesystem. This will be explicitly marked as an CONFIG_EXPERIMENTAL filesystem, and will in affect be a "development fork" of ext3. A similar split of the fs/jbd will be made in order to support 64-bit jbd, which will be used by fs/ext4 and future versions of ocfs2.
2) Bug fixes to fix 32-bit cleanliness issues, security/oops problems will go into fs/ext3, but all new development work will go into fs/ext4. There is some question about whether relatively low risk features such as slimming the extX in-core memory structure, and delayed allocation for ext3, which have no format impacts, should go into fs/ext3, or whether such enhancement should only benefit fs/ext4 users. This is a cost/benefit tradeoff for which the guidance of the LKML community about whether the loss in code stability is worth the improvements to current ext3 users, given the existence of the development branch.
In addition, we are assuming that various "low risk" changes that do involve format changes, such as support for higher resolution timestamps, will _not_ get integrated into the fs/ext3 codebase, and that people who want these features will have to use the stable/development fs/ext4 codebase.
3) The ext4 code base will continue to mount older ext3 filesystems, as this is necessary to ensure a future smooth upgrade path from ext3 to ext4 users. In addition, once a feature is added to the ext3dev filesystem, a huge amount of effort will be made to provide continuing support for the filesystem format enhancements going forward, just as we do with the syscall ABI. (Emergencies might happen if we make a major mistake and paint ourselves into a corner; but just as with changes to the kernel/userspace ABI, if there is some question about whether or not a particular filesystem format can be supported going forward indefinitely, we will not push changes into the mainline kernel until we are can be confident on this point.)
4) At some point, probably in 6-9 months when we are satisified with the set of features that have been added to fs/ext4, and confident that the filesystem format has stablized, we will submit a patch which causes the fs/ext4 code to register itself as the ext4 filesystem. The implementation may still require some shakedown before we are all confident that it is as stable as ext3 is today. At that point, perhaps 12-18 months out, we may request that the code in fs/ext3/*.c be deleted and that fs/ext4 register itself as supporting the ext3 filesystem as well.
We believe this should satisfy most of the concerns that were articulated, in particular those that Linus and Jeff were most concerned about. Comments are of course appreciated.
- Ted
Quoted 2 lines · Theodore Ts'o
To address these issues, after discussing the matter amongst ourselves, the ext2/3 developers would like to propose the following path forward.
Overall... ACK from me. Thanks for listening.
Quoted 9 lines
2) Bug fixes to fix 32-bit cleanliness issues,
security/oopsproblems will go into fs/ext3, but all new development work will go into fs/ext4. There is some question about whether relatively low risk features such as slimming the extX in-core memory structure, and delayed allocation for ext3, which have no format impacts, should go into fs/ext3, or whether such enhancement should only benefit fs/ext4 users. This is a cost/benefit tradeoff for which the guidance of the LKML community about whether the loss in code stability is worth the improvements to current ext3 users, given the existence of the development branch.
Agreed overall, though specifically for delayed allocation I think that's an ext4 thing:
* First off, I'm a big fan of delalloc, and (like extents) definitely want to see the feature implemented
* Delayed allocation, properly done, requires careful interaction with VM writeback (memory pressure or normal writeout), and may require some minor changes to generic code in fs/* and mm/*
* Delayed allocation changes I/O ordering, and may require some retuning for workloads to remain optimal
* Delayed allocation changes data layout on disk. HOPEFULLY for the better, but we won't know that until its been hammered a bit in the field.
So while I agree it has no format impacts, I also think it has a non-trivial -- and currently unknown -- impact on stable systems.
Also for the reasons listed, I think ext4 would be a far superior testbed for delalloc.
Quoted 5 lines
In addition, we are assuming that various "low risk" changes that do involve format changes, such as support for higher resolution timestamps, will _not_ get integrated into the fs/ext3 codebase, and that people who want these features will have to use the stable/development fs/ext4 codebase.
ACK
Quoted 11 lines
3) The ext4 code base will continue to mount older ext3 filesystems, as this is necessary to ensure a future smooth upgrade path from ext3 to ext4 users. In addition, once a feature is added to the ext3dev filesystem, a huge amount of effort will be made to provide continuing support for the filesystem format enhancements going forward, just as we do with the syscall ABI. (Emergencies might happen if we make a major mistake and paint ourselves into a corner; but just as with changes to the kernel/userspace ABI, if there is some question about whether or not a particular filesystem format can be supported going forward indefinitely, we will not push changes into the mainline kernel until we are can be confident on this point.)
ACK
Quoted 9 lines
4) At some point, probably in 6-9 months when we are satisified with the set of features that have been added to fs/ext4, and confident that the filesystem format has stablized, we will submit a patch which causes the fs/ext4 code to register itself as the ext4 filesystem. The implementation may still require some shakedown before we are all confident that it is as stable as ext3 is today. At that point, perhaps 12-18 months out, we may request that the code in
fs/ext3/*.cbe deleted and that fs/ext4 register itself as supporting the ext3 filesystem as well.
I continue to have a concern that it will become tougher over time to support all these features in the same codebase... so consider this a reluctant "ACK" for this last paragraph. :)
Jeff
Quoted 8 lines · Jeff Garzik
Agreed overall, though specifically for delayed allocation I think that's an ext4 thing:
* First off, I'm a big fan of delalloc, and (like extents) definitely want to see the feature implemented
* Delayed allocation, properly done, requires careful interaction with VM writeback (memory pressure or normal writeout), and may require some minor changes to generic code infs/*andmm/*
To be honest, I'd like to see more delayed allocation
infrastructure in the VFS itself. XFS has to maintain an entire chunk of state for it, and I suspect ext4 will as well. I'd love to get delayed allocation into OCFS2 someday. Why not move to where we can share the in-memory accounting code?
Now, we'd probably want to start by prototyping it in ext4
directly. Once it's stable as a filesystem feature, we can see where XFS and ext4 overlap, etc, etc. But I'd like to keep a more generic direction in mind.
Joel
Signature · 9 lines
"The suffering man ought really to consume his own smoke; there is no good in emitting smoke till you have made it into fire." - thomas carlyle Joel Becker Principal Software Developer Oracle E-mail: [email protected] Phone: (650) 506-8127
Quoted 10 lines · Joel Becker
Quoted 8 lines · Jeff Garzik
Agreed overall, though specifically for delayed allocation I think that's an ext4 thing:
* First off, I'm a big fan of delalloc, and (like extents) definitely want to see the feature implemented
* Delayed allocation, properly done, requires careful interaction with VM writeback (memory pressure or normal writeout), and may require some minor changes to generic code infs/*andmm/*
Quoted 9 lines
To be honest, I'd like to see more delayed allocation
infrastructure in the VFS itself. XFS has to maintain an entire chunk of state for it, and I suspect ext4 will as well. I'd love to get delayed allocation into OCFS2 someday. Why not move to where we can share the in-memory accounting code?
Now, we'd probably want to start by prototyping it in ext4
directly. Once it's stable as a filesystem feature, we can see where XFS and ext4 overlap, etc, etc. But I'd like to keep a more generic direction in mind.
I tried adding "delayed allocation" for ext3 earlier. Yes. VFS level infrastructure would be nice. But, I haven't found much that we can do at VFS - which is common across all the filesystems (except mpage_writepage(s) handling). Most of the stuff is specific to filesystem implementation (even though it could be common) - coming out with VFS level interfaces to suite all the different filesystem delalloc would be *interesting* exercise.
If you have ideas on what are the common/generic stuff we can do in VFS - I can take a look at it again :)
Thanks,
Badari
Quoted 7 lines · Badari Pulavarty
I tried adding "delayed allocation" for ext3 earlier. Yes. VFS level infrastructure would be nice. But, I haven't found much that we can do at VFS - which is common across all the filesystems (except
mpage_writepage(s)handling). Most of the stuff is specific to filesystem implementation (even though it could be common) - coming out with VFS level interfaces to suite all the different filesystem delalloc would be *interesting* exercise.
Well, to be fair, I'm just going by what little I know about
XFS. They maintain a cache of all pages waiting on delayed allocation for writepack. Why have this entire cache (hash, list, whatever) when we could create some state on in the pagecache? We save a large chunk of memory and some complex writeback code. I suspect you were thinking of this when you said "mpage_writepage(s) handling". But this is a large complexity win if we can do it.
The same with metadata/data ordering issues. ie, data=ordered
or even plain "creat(2); write(2)". I don't know how generic the ordering is for each filesystem, but there is always room for play.
On-disk, of course each filesystem is going to be different.
I'm not sure we could fit a fully-generic aops->reserve_space() & aops->commit_space() API. But I don't think we need to.
Joel
Signature · 8 lines
"Well-timed silence hath more eloquence than speech."
- Martin Fraquhar Tupper
Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127Quoted 8 lines · Joel Becker
Quoted 6 lines · Badari Pulavarty
I tried adding "delayed allocation" for ext3 earlier. Yes. VFS level
infrastructure would be nice. But, I haven't found much that we can
do at VFS - which is common across all the filesystems (exceptmpage_writepage(s)handling). Most of the stuff is specific to filesystem implementation (even though it could be common) - coming out with VFS level interfaces to suite all the different filesystem
delalloc would be *interesting* exercise.
Quoted 7 lines
Well, to be fair, I'm just going by what little I know about
XFS. They maintain a cache of all pages waiting on delayed allocation for writepack. Why have this entire cache (hash, list, whatever) when we could create some state on in the pagecache? We save a large chunk of memory and some complex writeback code. I suspect you were thinking of this when you said "mpage_writepage(s)handling". But this is a large complexity win if we can do it.
No, XFS does not do this, when it gets asked to write out a page which is delayed alloc, it goes and converts the delayed alloc extent to real disk space. It then uses the page cache/buffer heads to find all the contiguous pages which it can turn into a singe disk I/O. The code is made more complex by other possible states for the data. The only information internal to XFS though is its extent structures.
Steve
Quoted 9 lines · Joel Becker
Quoted 7 lines · Badari Pulavarty
I tried adding "delayed allocation" for ext3 earlier. Yes. VFS level infrastructure would be nice. But, I haven't found much that we can do at VFS - which is common across all the filesystems (except
mpage_writepage(s)handling). Most of the stuff is specific to filesystem implementation (even though it could be common) - coming out with VFS level interfaces to suite all the different filesystem delalloc would be *interesting* exercise.
Quoted 13 lines
Well, to be fair, I'm just going by what little I know about
XFS. They maintain a cache of all pages waiting on delayed allocation for writepack. Why have this entire cache (hash, list, whatever) when we could create some state on in the pagecache? We save a large chunk of memory and some complex writeback code. I suspect you were thinking of this when you said "mpage_writepage(s)handling". But this is a large complexity win if we can do it.
The same with metadata/data ordering issues. ie, data=ordered
or even plain "creat(2); write(2)". I don't know how generic the ordering is for each filesystem, but there is always room for play.
On-disk, of course each filesystem is going to be different.
I'm not sure we could fit a fully-genericaops->reserve_space()&aops->commit_space()API. But I don't think we need to.
Unfortunately, I haven't looked at XFS delalloc implementation indetail to understand what exact they would need from VFS (or could be pushed to VFS). I purely tried to work with current ext3 code and current VFS support. What I find is that -
1) Instead of allocating a block at prepare time, we need to be able to "reserve" a block (so it won't file as part of writeback). And, as part of writeback - we need a way to figure out if a given page did indeed really reserve the block. (we need to make sure the allocation succeeds for those). We might need a pageflag for this (but I haven't decided that its absolutely needed).
2) Needed a way to cluster bunch of (contig) pages and allocate disk blocksfor those in a single shot - which is NOT a direct delalloc requirement, but that is the whole reason for doing delalloc. (Suprana did few radix_tree interfaces for this).
Other than these general VFS level ones - I had to play with journal lock ordering issues (very specific to ext3 stuff). To work around the journalling issues, I had to do my own mpage_writepages() since the changes I need are specific to ext3 journalling - I am not sure if they are going to be useful for other filesystems or not.
If you can think of general infrastructure you need for OCFS2, please let me know - we can come with commonality.
Thanks,
Badari
Quoted 4 lines · Theodore Ts'o
1) The creation of a new filesystem codebase in the 2.6 kernel tree in
/usr/src/linux/fs/ext4that will initially register itself as the "ext3dev"
Why not call it ext4 from the beginning too? Calling the directory differently from the file system can only cause confusion.
I assume if it's marked very experimental people who value their data will avoid it for the time being.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 4 lines · Theodore Ts'o
1) The creation of a new filesystem codebase in the 2.6 kernel tree in
/usr/src/linux/fs/ext4that will initially register itself as the "ext3dev"Why not call it ext4 from the beginning too? Calling the directory differently from the file system can only cause confusion.
I assume if it's marked very experimental people who value their data will avoid it for the time being.
There were a lot of people who were concerned that simply marking it CONFIG_EXPERIMENTAL might not be enough for to make it very clear that the filesystem format is still changing. In order to address this concern, we want /etc/fstab to make it abundantly clear that the filesystem format itself is not necessarily stable, and that new features are being added that might not be supported on older kernels.
In general the same old rules will apply in that unless you explicitly enable a feature via tune2fs or at mke2fs-time, it should work on older kernels (just as in theory all of the kernel code that we are throwing willy-nilly into userpsace is **supposed** to be forward compatible), but if we paint ourselves in the a corner, there may be a reason why we have to break that compatibility promise.
- Ted
Quoted 13 lines · Theodore Tso
Quoted 11 lines · Andi Kleen
Quoted 4 lines · Theodore Ts'o
1) The creation of a new filesystem codebase in the 2.6 kernel tree in
/usr/src/linux/fs/ext4that will initially register itself as the "ext3dev"Why not call it ext4 from the beginning too? Calling the directory differently from the file system can only cause confusion.
I assume if it's marked very experimental people who value their data will avoid it for the time being.
Quoted 8 lines
There were a lot of people who were concerned that simply marking it
CONFIG_EXPERIMENTALmight not be enough for to make it very clear that the filesystem format is still changing. In order to address this concern, we want/etc/fstabto make it abundantly clear that the filesystem format itself is not necessarily stable, and that new features are being added that might not be supported on older kernels.
...
What about a dependency on CONFIG_BROKEN?
This will require everyone who wants to use it to manually edit the Kconfig file for removing the dependency - which sounds like a good idea.
Quoted 1 line
- Ted
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 9 lines · Adrian Bunk
Quoted 8 lines
There were a lot of people who were concerned that simply marking it
CONFIG_EXPERIMENTALmight not be enough for to make it very clear that the filesystem format is still changing. In order to address this concern, we want/etc/fstabto make it abundantly clear that the filesystem format itself is not necessarily stable, and that new features are being added that might not be supported on older kernels.
...
Quoted 5 lines
What about a dependency on
CONFIG_BROKEN?This will require everyone who wants to use it to manually edit the
Kconfigfile for removing the dependency - which sounds like a good idea.
It's not broken; just experimental. I think CONFIG_EXPERIMENTAL is accurately describes the status of fs/ext4 as we plan it. Patches will be reviewed and tested via quilt series and/or git trees before they get pushed to fs/ext4, so the expectation isn't that data will be lost; only that if you want to downgrade to an earlier kernel, and explicitly enable the latest bleeding edge feature, you might have to do some work first. (Which might be dump/reformat/restore, or it might be "tune2fs -O ^new_feature /dev/sdXX; e2fsck /dev/sdXX" with a development snapshot of e2fsprogs.)
- Ted
Quoted 6 lines · you
Given the recent discussion on LKML two weeks ago, it is clear that many people feel they have a stake in the future development plans of the ext2/ext3 filesystem, as it one of the most popular and commonly used filesystems, particular amongst the kernel development community. For this reason, the stakes are higher than it would be for other filesystems.
http://en.wikipedia.org/wiki/Ext4
Signature · 5 lines
Patrick McFarland || www.AdTerrasPerAspera.com "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 10 lines · Patrick McFarland
Quoted 6 lines · you
Given the recent discussion on LKML two weeks ago, it is clear that many people feel they have a stake in the future development plans of the ext2/ext3 filesystem, as it one of the most popular and commonly used filesystems, particular amongst the kernel development community. For this reason, the stakes are higher than it would be for other filesystems.
Thanks, another wiki page for this project is hosted at:
http://fedoraproject.org/wiki/ext3-devel
Current patch set is at
http://ext2.sourceforge.net/48bitext3/patches/latest/
and current e2fsprogs patch set is at
http://ext2.sourceforge.net/48bitext3/patches/e2fsprogs/48bit-
e2fsprogs-1.39/
Quoted 1 line · Theodore Ts'o
3) The ext4 code base will continue to mount older ext3 filesystems,
Everybody seems enthusiastic about this plan so I must have missed something, but
how will this compatibility prevent the ext4 code from looking like: 'if (ext4) { ... } else { /* ext3 */ ... }'?
Regards.
Signature · 1 line
Guillaume
Quoted 5 lines · Guillaume Chazarain
Quoted 1 line · Theodore Ts'o
3) The ext4 code base will continue to mount older ext3 filesystems,
Everybody seems enthusiastic about this plan so I must have missed something, but
how will this compatibility prevent the ext4 code from looking like: 'if (ext4) { ... } else { /* ext3 */ ... }'?
It doesn't. But you can't win them all...
Jeff