Ext4 devel interlock meeting minutes (May 7, 2007)
Ext4 Developer Interlock Call: 05/07/2007 Meeting Minutes
Attendees: Mingming Cao, Dave Kleikamp, Ted Ts'o, Aneesh Veetil, Takashi Sato, Jose Santos, Avantika Mathur,
Minutes can be accessed at: http://ext4.wiki.kernel.org/index.php/Ext4_Developer%27s_Conference_Call
Discussed pushing ext4 patches upstream.
- Preallocation Patches: These patches are almost ready to push, once we have consensus from all architecture maintainers on the format of the system call. Mingming will help with this.
- Nanosecond Timestamps: Ted will take a closer look at this patch before pushing
- It was proposed to push everything in the ext4 git tree up until the unstable patches. Before pushing these patches, we must run tests on the patches and use the check-patches scripts to perform sanity checking.
e2fsprogs:
- Ted recently posted e2fsprogs-1.39-tyt3; which includes patches supporting extents. This is a work in progress.
- Jose used the patches to e2fsprogs sent by Valerie Clement to create large filesystems (with >32-bit block numbers)
-- Ted suggested that these patches be added to e2fsprogs-tyt3, with
a flag set when in use, indicating that no shared libraries should be used.
- Ted is planning to discontinue use of Mercurial for e2fsprogs, and switch entirely to git. By using git, we can have a stable and development branch for e2fsprogs. This will also reduce the work Ted does to create a patch series.
Migration:
- Aneesh looked at the online defragmentation code, and feels it may not be usable for migration because it is solely extents based. Mingming pointed out that the new defrag patches also have support for indirect mapped blocks, Aneesh will look into these new patches.
- Ted suggested doing the migration through two ioctls, the first migrating the inode online via ioctl, and the second using the online-defrag ioctl.
Metablock Groups:
- Mingming mentioned that in the metablock group feature, the inode metadata is not moved with the rest of the metadata. Ted will check this, as he thought this had been implemented.
- Online resize does not use metablock groups, and is currently limited to 2TB.
- Also need to remove sanity checks in the mount code
- Once these issues are fixed, the metablock group feature can be turned on by default for ext4 filesysytems
- The extents option will be turned on by default for all ext4 filesystems with the extents feature enabled.
Quoted 6 lines · Avantika Mathur
Metablock Groups:
- Mingming mentioned that in the metablock group feature, the inode metadata is not moved with the rest of the metadata. Ted will check this, as he thought this had been implemented.
Quoted 7 lines
- Online resize does not use metablock groups, and is currently limited to 2TB.
- Also need to remove sanity checks in the mount code
- Once these issues are fixed, the metablock group feature can be turned on by default for ext4 filesysytems
In my recent investigation of META_BG, it appears that BOTH the kernel and e2fsprogs do NOT handle group metadata (bitmaps, itable) outside the group.
In the kernel ext[234]_check_descriptors() requires the bitmaps and itable to be inside the group even though META_BG is supported for a long time.
In e2fsck/super.c check_super_block() also requires that the bitmaps and itable be inside the group even though e2fsck has claimed META_BG for a long time.
That means it is virtually impossible to allow the group metadata to move under the META_BG feature. It might be possible to predicate this change under both 64BIT and META_BG, though this is somewhat non-standard.
Alternately, we could allow the relocation of bitmaps and itables under BIG_BG, which is itself covers the relocation aspects and also allows the ability to change the group size.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
Quoted 22 lines · Andreas Dilger
Quoted 6 lines · Avantika Mathur
Metablock Groups:
- Mingming mentioned that in the metablock group feature, the inode metadata is not moved with the rest of the metadata. Ted will check this, as he thought this had been implemented.
Quoted 7 lines
- Online resize does not use metablock groups, and is currently limited to 2TB.
- Also need to remove sanity checks in the mount code
- Once these issues are fixed, the metablock group feature can be turned on by default for ext4 filesysytems
In my recent investigation of
META_BG, it appears that BOTH the kernel and e2fsprogs do NOT handle group metadata (bitmaps, itable) outside the group.In the kernel ext[234]_check_descriptors() requires the bitmaps and itable to be inside the group even though
META_BGis supported for a long time.
Quoted 6 lines
In
e2fsck/super.ccheck_super_block()also requires that the bitmaps and itable be inside the group even though e2fsck has claimedMETA_BGfor a long time.That means it is virtually impossible to allow the group metadata to move under the
META_BGfeature.
Understand it's not straightforward to turn on this feature by default. Just try to understand, why the kernel and e2fsprogs issues you mentioned above are not possible to fix to support full META_BG feature?
Quoted 7 lines
It might be possible to predicate this change
under both 64BIT andMETA_BG, though this is somewhat non-standard.Alternately, we could allow the relocation of bitmaps and itables under
BIG_BG, which is itself covers the relocation aspects and also allows the ability to change the group size.
Well, I think the previous discussion about BIG_BG feature leads us to believe that turn on META_BG feature is good enough to support larger fs and reduce fragmentation(which is the new BIG_BG feature is trying to address). After all the relocation of metadata (bitmaps, itables and group descriptors) is what META_BG feature means.
Regards,
Mingming
Quoted 8 lines · Mingming Cao
Quoted 2 lines · Andreas Dilger
That means it is virtually impossible to allow the group metadata to move under the
META_BGfeature.Understand it's not straightforward to turn on this feature by default. Just try to understand, why the kernel and e2fsprogs issues you mentioned above are not possible to fix to support full
META_BGfeature?
Because older kernels and e2fsprogs will think they understand META_BG, and will consider the filesystem corrupt due to "badly placed" bitmaps and itable. That is exactly what FEATURE flags are supposed to protect against, so we can't change the semantics of the META_BG feature at this time. I understand that we WANTED to have META_BG have different semantics, but the implementation didn't follow the design for this flag.
Quoted 8 lines
Quoted 7 lines
It might be possible to predicate this change
under both 64BIT andMETA_BG, though this is somewhat non-standard.Alternately, we could allow the relocation of bitmaps and itables under
BIG_BG, which is itself covers the relocation aspects and also allows the ability to change the group size.
Quoted 5 lines
Well, I think the previous discussion about
BIG_BGfeature leads us to believe that turn onMETA_BGfeature is good enough to support larger fs and reducefragmentation(which is the new BIG_BG feature is trying to address). After all the relocation of metadata (bitmaps, itables and group descriptors) is whatMETA_BGfeature means.
That is what we WANTED the META_BG feature to mean. What was implemented in both ext2/3/4 and e2fsck is that META_BG means "group descriptor backups are in 1st, 2nd, last group in metagroup" and nothing else. At least the code is consistent between the two.
So that leaves us in the position that we either need to create a new INCOMPAT feature flag to allow relocation of bitmaps/itable, OR we need to make this conditional upon both META_BG and 64BIT being set. The latter is a bit confusing and is subject to error if only one or the other flag is checked by accident. If we make a new INCOMPAT flag then could almost as easily implement BIG_BG and give us more flexibility for the future, though I haven't looked at that code in a while.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.