ext4: ext4_split_extent shoult take care about extent zeroout v3
PATCH 1/5 ext4: ext4_split_extent shoult take care about extent zeroout v3
When ext4_split_extent_at() ends up doing zeroout & conversion to initialized instead of split & conversion, ext4_split_extent() gets confused and can wrongly mark the extent back as uninitialized resulting in end IO code getting confused from large unwritten extents and may result in data loss.
The example of problematic behavior is:
lblk len lblk len
ext4_split_extent() (ex=[1000,30,uninit], map=[1010,10])
ext4_split_extent_at() (split [1000,30,uninit] at 1020)
ext4_ext_insert_extent() -> ENOSPC
ext4_ext_zeroout()
-> extent [1000,30] is now initialized
ext4_split_extent_at() (split [1000,30,init] at 1010,
MARK_UNINIT1 | MARK_UNINIT2)
-> extent is split and parts marked as uninitializedFix the problem by rechecking extent type after the first ext4_split_extent_at() returns. None of split_flags can not be applied to initialized extent so this patch also add BUG_ON to prevent similar issues in future.
TESTCASE: https://github.com/dmonakhov/xfstests/commit/b8a55eb5ce28c6ff29e620ab090902fcd5833597
Changes since V2: Patch no longer depends on Jan's "disable-uninit-ext-mergring" patch.
--- fs/ext4/extents.c | 22 ++++++++++++++++------ 1 files changed, 16 insertions(+), 6 deletions(-)
fs/ext4/extents.c +16 -6
@@ -2943,6 +2943,10 @@ static int ext4_split_extent_at(handle_t *handle, newblock = split - ee_block +ext4_ext_pblock (ex);BUG_ON (split < ee_block || split >= (ee_block + ee_len)); +BUG_ON (!ext4_ext_is_uninitialized (ex) && + split_flag & (EXT4_EXT_MAY_ZEROOUT | +EXT4_EXT_MARK_UNINIT1 | +EXT4_EXT_MARK_UNINIT2 )); err =ext4_ext_get_access (handle, inode, path + depth);if (err)
@@ -3061,19 +3065,25 @@ static int ext4_split_extent(handle_t *handle,if (err) goto out; } - +/* +* Update path is required because previous ext4_split_extent_at() may +* result in split of original leaf or extent zeroout. +*/ ext4_ext_drop_refs (path); path =ext4_ext_find_extent (inode, map->m_lblk, path);if (IS_ERR (path))return PTR_ERR (path); + depth =ext_depth (inode); + ex = path[depth].p_ext; + uninitialized =ext4_ext_is_uninitialized (ex); + split_flag1 =0 ;if (map->m_lblk >= ee_block) { - split_flag1 = split_flag & (EXT4_EXT_MAY_ZEROOUT | -EXT4_EXT_DATA_VALID2 ); -if (uninitialized) + split_flag1 = split_flag &EXT4_EXT_DATA_VALID2 ; +if (uninitialized) { split_flag1 |=EXT4_EXT_MARK_UNINIT1 ; -if (split_flag &EXT4_EXT_MARK_UNINIT2 ) - split_flag1 |=EXT4_EXT_MARK_UNINIT2 ; + split_flag1 |= split_flag & (EXT4_EXT_MAY_ZEROOUT |EXT4_EXT_MARK_UNINIT2 ); + } err =ext4_split_extent_at (handle, inode, path, map->m_lblk, split_flag1, flags);if (err)
Signature · 1 line
1.7.1
PATCH 2/5 ext4: disable merging of uninitialized extents
Derived from Jan's patch:http://permalink.gmane.org/gmane.comp.file-systems.ext4/36470
Merging of uninitialized extents creates all sorts of interesting race possibilities when writeback / DIO races with fallocate. Thus ext4_convert_unwritten_extents_endio() has to deal with a case where extent to be converted needs to be split out first. That isn't nice for two reasons:
1) It may need allocation of extent tree block so ENOSPC is possible.
2) It complicates end_io handling code
So we disable merging of uninitialized extents which allows us to simplify the code. Extents will get merged after they are converted to initialized ones.
--- fs/ext4/extents.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)
fs/ext4/extents.c +5 -3
@@ -1584,10 +1584,12 @@ ext4_can_extents_be_merged(struct inode *inode, struct ext4_extent *ex1,unsigned short ext1_ee_len, ext2_ee_len, max_len;/* -* Make sure that either both extents are uninitialized, or -* both are _not_. +* Make sure that both extents are initialized. We don't merge +* uninitialized extents so that we can be sure that end_io code has +* the extent that was written properly split out and conversion to +* initialized is trivial. */ -if (ext4_ext_is_uninitialized (ex1) ^ext4_ext_is_uninitialized (ex2)) +if (ext4_ext_is_uninitialized (ex1) ||ext4_ext_is_uninitialized (ex2))return 0 ;if (ext4_ext_is_uninitialized (ex1))
Signature · 1 line
1.7.1
Re: [PATCH 2/5] ext4: disable merging of uninitialized extents
Quoted 8 lines · Dmitry Monakhov
Derived from Jan's patch:http://permalink.gmane.org/gmane.comp.file-systems.ext4/36470
Merging of uninitialized extents creates all sorts of interesting race possibilities when writeback / DIO races with fallocate. Thus
ext4_convert_unwritten_extents_endio()has to deal with a case where extent to be converted needs to be split out first. That isn't nice for two reasons:
Quoted 6 lines
1) It may need allocation of extent tree block so ENOSPC is possible.
2) It complicatesend_iohandling codeSo we disable merging of uninitialized extents which allows us to simplify the code. Extents will get merged after they are converted to initialized ones.
Looks good. You can add:
Honza
Quoted 28 lines
Signed-off-by Dmitry Monakhov <[email protected]>--- fs/ext4/extents.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)fs/ext4/extents.c +5
-3@@ -1584,10 +1584,12 @@ ext4_can_extents_be_merged(struct inode *inode, struct ext4_extent *ex1,unsigned short ext1_ee_len, ext2_ee_len, max_len;/* -* Make sure that either both extents are uninitialized, or -* both are _not_. +* Make sure that both extents are initialized. We don't merge +* uninitialized extents so that we can be sure that end_io code has +* the extent that was written properly split out and conversion to +* initialized is trivial. */ -if (ext4_ext_is_uninitialized (ex1) ^ext4_ext_is_uninitialized (ex2)) +if (ext4_ext_is_uninitialized (ex1) ||ext4_ext_is_uninitialized (ex2))return 0 ;if (ext4_ext_is_uninitialized (ex1))Signature · 1 line
1.7.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 2/5] ext4: disable merging of uninitialized extents
In reply to Dmitry MonakhovQuoted 8 lines · Dmitry Monakhov
Derived from Jan's patch:http://permalink.gmane.org/gmane.comp.file-systems.ext4/36470
Merging of uninitialized extents creates all sorts of interesting race possibilities when writeback / DIO races with fallocate. Thus
ext4_convert_unwritten_extents_endio()has to deal with a case where extent to be converted needs to be split out first. That isn't nice for two reasons:
Quoted 8 lines
1) It may need allocation of extent tree block so ENOSPC is possible.
2) It complicatesend_iohandling codeSo we disable merging of uninitialized extents which allows us to simplify the code. Extents will get merged after they are converted to initialized ones.
Signed-off-by Dmitry Monakhov <[email protected]>
After applied this patch, xfstests #275 will print a warning message.
kernel:EXT4-fs (sda2): failed to convert unwritten extents to written extents -- potential data loss! (inode 13, offset 1537212416, size 524288, error -28)
But IMHO we don't need to worry about it because it is hard to be trigger. I hit it because I run xfstests #275 several times. So just a note here.
Regards,
- Zheng
PATCH 3/5 ext4: add warning to ext4_convert_unwritten_extents_endio
In reply to Dmitry MonakhovSplitting extents inside endio is bad thing, but unfortunetly it is still possible. In fact we are pretty close to the moment when all related issues will be fixed. Let's warn developer if it still the case.
--- fs/ext4/extents.c | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-)
fs/ext4/extents.c +12 -1
@@ -3386,8 +3386,19 @@ static int ext4_convert_unwritten_extents_endio(handle_t *handle,"block %llu, max_blocks %u\n" , inode->i_ino, (unsigned long long)ee_block, ee_len); -/* If extent is larger than requested then split is required */ +/* If extent is larger than requested it is a clear sign that we still +* have some extent state machine issues left. So extent_split is still +* required. +* TODO: Once all related issues will be fixed this situation should be +* illegal. +*/ if (ee_block != map->m_lblk || ee_len > map->m_len) { +#ifdef EXT4_DEBUG +ext4_warning ("Inode (%ld) finished: extent logical block %llu," +" len %u; IO logical block %llu, len %u\n" , + inode->i_ino, (unsigned long long )ee_block, ee_len, + (unsigned long long )map->m_lblk, map->m_len); +#endif err =ext4_split_unwritten_extents (handle, inode, map, path,EXT4_GET_BLOCKS_CONVERT );if (err <0 )
Signature · 1 line
1.7.1
Re: [PATCH 3/5] ext4: add warning to ext4_convert_unwritten_extents_endio
Quoted 5 lines · Dmitry Monakhov
Splitting extents inside endio is bad thing, but unfortunetly it is still possible. In fact we are pretty close to the moment when all related issues will be fixed. Let's warn developer if it still the case.
Signed-off-by Dmitry Monakhov <[email protected]>
Looks good. You can add:
Honza
Quoted 32 lines
--- fs/ext4/extents.c | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-)fs/ext4/extents.c +12
-1@@ -3386,8 +3386,19 @@ static int ext4_convert_unwritten_extents_endio(handle_t *handle,"block %llu, max_blocks %u\n" , inode->i_ino, (unsigned long long)ee_block, ee_len); -/* If extent is larger than requested then split is required */ +/* If extent is larger than requested it is a clear sign that we still +* have some extent state machine issues left. So extent_split is still +* required. +* TODO: Once all related issues will be fixed this situation should be +* illegal. +*/ if (ee_block != map->m_lblk || ee_len > map->m_len) { +#ifdef EXT4_DEBUG +ext4_warning ("Inode (%ld) finished: extent logical block %llu," +" len %u; IO logical block %llu, len %u\n" , + inode->i_ino, (unsigned long long )ee_block, ee_len, + (unsigned long long )map->m_lblk, map->m_len); +#endif err =ext4_split_unwritten_extents (handle, inode, map, path,EXT4_GET_BLOCKS_CONVERT );if (err <0 )Signature · 1 line
1.7.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 3/5] ext4: add warning to ext4_convert_unwritten_extents_endio
In reply to Dmitry MonakhovQuoted 17 lines · Dmitry Monakhov
Splitting extents inside endio is bad thing, but unfortunetly it is still possible. In fact we are pretty close to the moment when all related issues will be fixed. Let's warn developer if it still the case.
Signed-off-by Dmitry Monakhov <[email protected]>--- fs/ext4/extents.c | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-)fs/ext4/extents.c +0
-0@@ -3386,8 +3386,19 @@ static int ext4_convert_unwritten_extents_endio(handle_t *handle,"block %llu, max_blocks %u\n" , inode->i_ino, (unsigned long long)ee_block, ee_len);
fs/ext4/extents.c static int ext4_convert_unwritten_extents_endio(handle_t *handle, · line 3397
Quoted 10 lines
fs/ext4/extents.c +8
-1-/* If extent is larger than requested then split is required */ +/* If extent is larger than requested it is a clear sign that we still +* have some extent state machine issues left. So extent_split is still +* required. +* TODO: Once all related issues will be fixed this situation should be +* illegal. +*/ if (ee_block != map->m_lblk || ee_len > map->m_len) { +#ifdef EXT4_DEBUG + ext4_warning("Inode (%ld) finished: extent logical block %llu,"
A parameter is missing.
fs/ext4/extents.c +0 -0
@@ -3395,7 +3395,8 @@ static int
ext4_convert_unwritten_extents_endio(handle_t *handle,
*/
if (ee_block != map->m_lblk || ee_len > map->m_len) { #ifdef EXT4_DEBUG
- ext4_warning("Inode (%ld) finished: extent logical block %llu,"
+ ext4_warning(inode->i_sb,
+ "Inode (%ld) finished: extent logical block %llu,"
" len %u; IO logical block %llu, len %u\n",
inode->i_ino, (unsigned long long)ee_block, ee_len,
(unsigned long long)map->m_lblk, map->m_len);Regards,
- Zheng
Quoted 14 lines
fs/ext4/extents.c +4
-0+" len %u; IO logical block %llu, len %u\n" , + inode->i_ino, (unsigned long long )ee_block, ee_len, + (unsigned long long)map->m_lblk, map->m_len); +#endif err =ext4_split_unwritten_extents (handle, inode, map, path,EXT4_GET_BLOCKS_CONVERT );if (err <0 )Signature · 6 lines
1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 4/5 ext4: remove unnecessary wait for extent conversion in ext4_fallocate()
In reply to Dmitry MonakhovFrom: Jan Kara <[email protected]>
Now that we don't merge uninitialized extents anymore, ext4_fallocate() is free to operate on the inode while there are still some extent conversions pending - it won't disturb them in any way.
--- fs/ext4/extents.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
fs/ext4/extents.c +0 -2
@@ -4391,8 +4391,6 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)if (len <=EXT_UNINIT_MAX_LEN << blkbits) flags |=EXT4_GET_BLOCKS_NO_NORMALIZE ; -/* Prevent race condition between unwritten */ -ext4_flush_unwritten_io (inode); retry:while (ret >=0 && ret < max_blocks) { map.m_lblk = map.m_lblk + ret;
Signature · 1 line
1.7.1
PATCH 5/5 ext4: invalidate exntent-status-tree during extent_migration
In reply to Dmitry Monakhovmext_replace_branches() will change inode's extents layout so we have to drop corresponding cache.
TESTCASE: 301'th xfstest was not yet accepted to official xfstest's branch
and can be found here: https://github.com/dmonakhov/xfstests/commit/7b7efeee30a41109201e2040034e71db9b66ddc0
--- fs/ext4/move_extent.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
fs/ext4/move_extent.c +8 -0
@@ -666,6 +666,14 @@ mext_replace_branches(handle_t *handle, struct inode *orig_inode,int replaced_count =0 ;int dext_alen; + *err =ext4_es_remove_extent (orig_inode, from, count); +if (*err) + goto out; + + *err =ext4_es_remove_extent (donor_inode, from, count); +if (*err) + goto out; +/* Get the original extent for the block "orig_off" */ *err =get_ext_path (orig_inode, orig_off, &orig_path);if (*err)
Signature · 1 line
1.7.1
Re: [PATCH 5/5] ext4: invalidate exntent-status-tree during extent_migration
Quoted 5 lines · Dmitry Monakhov
mext_replace_branches()will change inode's extents layout so we have to drop corresponding cache.TESTCASE: 301'th xfstest was not yet accepted to official xfstest's branch
and can be found here: https://github.com/dmonakhov/xfstests/commit/7b7efeee30a41109201e2040034e71db9b66ddc0
One BIG note about this patch.
It fix BUG_ON(bh->b_blocknr != pblock) in fs/ext4/inode.c:1452 but 300'th xfstest(from my repo) still failed due to data corruption in verifier thread.
LOG:MOUNT_OPTIONS -- -o acl,user_xattr /dev/mapper/vzvg-scratch_dev /mnt_scratch
300 33s ... [failed, exit status 1] - output mismatch (see 300.out.bad)
--- 300.out 2013-02-20 12:46:24.000000000 +0400
+++ 300.out.bad 2013-02-25 20:18:24.000000000 +0400
@@ -2,3 +2,5 @@
Start defragment activity
+failed: '/usr/local/bin/fio /tmp/1665-300.fio'
+(see 300.full for details)
...
(Run 'diff -u 300.out 300.out.bad' to see the entire diff)#300.full
crc32c: verify failed at file /mnt_scratch/test2 offset 16973824, length
65536
Expected CRC: d062565e
Received CRC: f2cd2028
received data dumped as test2.16973824.received
expected data dumped as test2.16973824.expecteddefrag-4k: Laying out IO file(s) (1 file(s) / 3400MB)
donor-file-fuzzer: Laying out IO file(s) (1 file(s) / 3400MB)
#DIFF:
--- /tmp/test2.16973824.expected 2013-02-25 20:23:04.000000000
+0400
+++ /tmp/test2.16973824.received 2013-02-25 20:22:55.000000000
+0400@@ -254,3844 +254,6 @@
0000fd0 eb95 6201 89ec 151a bd72 9675 203c 0d80
0000fe0 b7ae b415 b8cc 0b2f b6f5 baab 9d0e 1741
0000ff0 f6de fbda d64b 1bd3 5edb 040c 7cdc 18e6
-0001000 0bdb 5114 cd95 01eb 017b a435 d128 11af
-0001010 202f 93c9 a80a 013e a405 c261 8b1c 0dab
-0001020 b480 c649 1032 1b0a 3690 7e89 dee1 0ac7
-0001030 26d2 9489 3c97 0bd8 24da 5f28 3d4e 066d
-0001040 049b b978 7815 159c 8093 b4e1 b246 1c25
.....
-000ffc0 bbb1 fa68 5622 022d 9776 0174 2d90 1ecb
-000ffd0 92ee b473 64f7 0bcb 725d 2d17 9265 0fff
-000ffe0 6e4b ec74 5bc0 0618 0dc9 e669 953d 002f
-000fff0 a1b9 35e8 ff73 17a5 9437 15e0 24fa 150a
+0001000 0000 0000 0000 0000 0000 0000 0000 0000
+*
0010000
It seems that we data was written to uninitialized extent, but unwritten->initialized extent conversion was missed somewhere. I have not fix for that issue yet.
Quoted 28 lines
Signed-off-by Dmitry Monakhov <[email protected]>--- fs/ext4/move_extent.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)fs/ext4/move_extent.c +8
-0@@ -666,6 +666,14 @@ mext_replace_branches(handle_t *handle, struct inode *orig_inode,int replaced_count =0 ;int dext_alen; + *err =ext4_es_remove_extent (orig_inode, from, count); +if (*err) + goto out; + + *err =ext4_es_remove_extent (donor_inode, from, count); +if (*err) + goto out; +/* Get the original extent for the block "orig_off" */ *err =get_ext_path (orig_inode, orig_off, &orig_path);if (*err)Signature · 1 line
1.7.1
Re: [PATCH 5/5] ext4: invalidate exntent-status-tree during extent_migration
Hi Dmitry,
Thank you very much for fixing this regression. A very quick test is run in my desktop and xfstests #300 and #301 can pass. Feel free to add:
I will add self-testing infrastructure in status tree ASAP and re-run xfstests.
Thanks!!
- Zheng
Quoted 94 lines · Dmitry Monakhov
Quoted 5 lines · Dmitry Monakhov
mext_replace_branches()will change inode's extents layout so we have to drop corresponding cache.TESTCASE: 301'th xfstest was not yet accepted to official xfstest's branch
and can be found here: https://github.com/dmonakhov/xfstests/commit/7b7efeee30a41109201e2040034e71db9b66ddc0One BIG note about this patch.
It fixBUG_ON(bh->b_blocknr != pblock)infs/ext4/inode.c:1452but 300'th xfstest(from my repo) still failed due to data corruption in verifier thread.
LOG:MOUNT_OPTIONS-- -o acl,user_xattr/dev/mapper/vzvg-scratch_dev/mnt_scratch300 33s ... [failed, exit status 1] - output mismatch (see 300.out.bad)
--- 300.out 2013-02-20 12:46:24.000000000 +0400 +++ 300.out.bad 2013-02-25 20:18:24.000000000 +0400 @@ -2,3 +2,5 @@ Start defragment activity +failed: '/usr/local/bin/fio /tmp/1665-300.fio' +(see 300.full for details) ... (Run 'diff -u 300.out 300.out.bad' to see the entire diff)#300.full
crc32c: verify failed at file /mnt_scratch/test2 offset 16973824, length65536 Expected CRC: d062565e Received CRC: f2cd2028 received data dumped as test2.16973824.received expected data dumped as test2.16973824.expecteddefrag-4k: Laying out IO file(s) (1 file(s) / 3400MB)
donor-file-fuzzer: Laying out IO file(s) (1 file(s) / 3400MB)#DIFF:
---/tmp/test2.16973824.expected2013-02-25 20:23:04.000000000+0400 +++ /tmp/test2.16973824.received 2013-02-25 20:22:55.000000000 +0400(quoted diff) +2
-9@@ -254,3844 +254,6 @@ 0000fd0 eb95 6201 89ec 151a bd72 9675 203c 0d80 0000fe0 b7ae b415 b8cc 0b2f b6f5 baab 9d0e 1741 0000ff0 f6de fbda d64b 1bd3 5edb 040c 7cdc 18e6 -0001000 0bdb 5114 cd95 01eb 017b a435 d128 11af -0001010 202f 93c9 a80a 013e a405 c261 8b1c 0dab -0001020 b480 c649 1032 1b0a 3690 7e89 dee1 0ac7 -0001030 26d2 9489 3c97 0bd8 24da 5f28 3d4e 066d -0001040 049b b978 7815 159c 8093 b4e1 b246 1c25 -000ffc0 bbb1 fa68 5622 022d 9776 0174 2d90 1ecb -000ffd0 92ee b473 64f7 0bcb 725d 2d17 9265 0fff -000ffe0 6e4b ec74 5bc0 0618 0dc9 e669 953d 002f -000fff0 a1b9 35e8 ff73 17a5 9437 15e0 24fa 150a +0001000 0000 0000 0000 0000 0000 0000 0000 0000 +* 0010000It seems that we data was written to uninitialized extent, but
unwritten->initializedextent conversion was missed somewhere. I have not fix for that issue yet.Quoted 28 lines
Signed-off-by Dmitry Monakhov <[email protected]>--- fs/ext4/move_extent.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)fs/ext4/move_extent.c +8
-0@@ -666,6 +666,14 @@ mext_replace_branches(handle_t *handle, struct inode *orig_inode,int replaced_count =0 ;int dext_alen; + *err =ext4_es_remove_extent (orig_inode, from, count); +if (*err) + goto out; + + *err =ext4_es_remove_extent (donor_inode, from, count); +if (*err) + goto out; +/* Get the original extent for the block "orig_off" */ *err =get_ext_path (orig_inode, orig_off, &orig_path);if (*err)Signature · 1 line
1.7.1--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 1/1 ext4: fix wrong m_len value after unwritten extent conversion (Re: [PATCH 5/5] ext4: invalidate...)
In reply to Dmitry MonakhovHi Dmitry and Ted,
On Mon, Feb 25, 2013 at 08:29:46PM +0400, Dmitry Monakhov wrote:
[snip]
Quoted 53 lines
One BIG note about this patch.
It fixBUG_ON(bh->b_blocknr != pblock)infs/ext4/inode.c:1452but 300'th xfstest(from my repo) still failed due to data corruption in verifier thread.
LOG:MOUNT_OPTIONS-- -o acl,user_xattr/dev/mapper/vzvg-scratch_dev/mnt_scratch300 33s ... [failed, exit status 1] - output mismatch (see 300.out.bad)
--- 300.out 2013-02-20 12:46:24.000000000 +0400 +++ 300.out.bad 2013-02-25 20:18:24.000000000 +0400 @@ -2,3 +2,5 @@ Start defragment activity +failed: '/usr/local/bin/fio /tmp/1665-300.fio' +(see 300.full for details) ... (Run 'diff -u 300.out 300.out.bad' to see the entire diff)#300.full
crc32c: verify failed at file /mnt_scratch/test2 offset 16973824, length65536 Expected CRC: d062565e Received CRC: f2cd2028 received data dumped as test2.16973824.received expected data dumped as test2.16973824.expecteddefrag-4k: Laying out IO file(s) (1 file(s) / 3400MB)
donor-file-fuzzer: Laying out IO file(s) (1 file(s) / 3400MB)#DIFF:
---/tmp/test2.16973824.expected2013-02-25 20:23:04.000000000+0400 +++ /tmp/test2.16973824.received 2013-02-25 20:22:55.000000000 +0400(quoted diff) +2
-9@@ -254,3844 +254,6 @@ 0000fd0 eb95 6201 89ec 151a bd72 9675 203c 0d80 0000fe0 b7ae b415 b8cc 0b2f b6f5 baab 9d0e 1741 0000ff0 f6de fbda d64b 1bd3 5edb 040c 7cdc 18e6 -0001000 0bdb 5114 cd95 01eb 017b a435 d128 11af -0001010 202f 93c9 a80a 013e a405 c261 8b1c 0dab -0001020 b480 c649 1032 1b0a 3690 7e89 dee1 0ac7 -0001030 26d2 9489 3c97 0bd8 24da 5f28 3d4e 066d -0001040 049b b978 7815 159c 8093 b4e1 b246 1c25 -000ffc0 bbb1 fa68 5622 022d 9776 0174 2d90 1ecb -000ffd0 92ee b473 64f7 0bcb 725d 2d17 9265 0fff -000ffe0 6e4b ec74 5bc0 0618 0dc9 e669 953d 002f -000fff0 a1b9 35e8 ff73 17a5 9437 15e0 24fa 150a +0001000 0000 0000 0000 0000 0000 0000 0000 0000 +* 0010000
Quoted 3 lines
It seems that we data was written to uninitialized extent, but
unwritten->initializedextent conversion was missed somewhere. I have not fix for that issue yet.
I take a close look at this bug, and I found that a wrong 'map->m_len' is returned from ext4_ext_map_blocks when we try to convert an unwritten extent with EXT4_GET_BLOCKS_IO_CONVERT_EXT flag. Here we always assume that the return value of ext/ind_map_blocks() is equal to m_len. But here it breaks this assumption. Let us consider what happens.
When we do a dio write for a extent-based file, we will preallocate an unwritten extent for it. After dio write has been done, we will convert it in ext4_end_io callback. We use ext4_convert_unwritten_extents to do this conversion. Then the codepath is like beblow:
ext4_convert_unwritten_extents()
->ext4_map_blocks()
->ext4_es_lookup_extent()
[We can lookup an unwritten extent]
->ext4_ext_map_blocks()
->ext4_ext_handle_uninitialized_extents()
->ext4_es_insert_extent()
[We update status tree, but the length of written extent is wrong.
The length of written extent is greater than the length of we
have converted]The following case demonstrates what happens.
1. We do a dio write. An unwritten extent will be preallocated.
status tree: [0, 23]:unwritten
extent tree: [0, 23]:unwritten
2. We try to convert an unwritten extent, e.g. [0, 15]. But due to this bug we will update all unwritten extent in status tree.
status tree: [0, 23]:written
extent tree: [0, 15]:written, [16, 23]:unwritten
3. Then we try to convert the following unwritten extent, but we will return directly in ext4_map_blocks because we lookup an written extent from status tree, and that unwritten extent will never be converted. At the time if e4defrag is running, the status tree could be removed, and we could read an unwritten extent.
Certainly this is only my *guess* because in my sandbox xfstests #300 and #301 never fail. I am not sure this patch could fix this regression. *But* I am pretty sure it will cause some potential bugs if it hasn't been fixed. I paste the patch at the below, which has been tested by xfstests plus Dmitry's #300 and #301 test case at the following configurations.
* Ext4 4k block
* Ext4 4k block w/dioread_nolock
* EXt4 4k block w/bigalloc
Dmitry, I really appreciate if you could give this patch a try. Hope it can fix this regression. Please let me know if there is any update. Thank you very much.
Regards,
- Zheng
Subject: [PATCH] ext4: fix wrong m_len value after unwritten extent conversion
From: Zheng Liu <[email protected]>
We always assume that the return value of ext4_ext_map_blocks is equal to map->m_len but when we try to convert an unwritten extent 'm_len' value will break this assumption. It is harmless until we use status tree as a extent cache because we need to update status tree according to 'm_len' value.
Meanwhile this commit marks EXT4_MAP_MAPPED flag after unwritten extent conversion. It shouldn't cause a bug because we update status tree according to checking EXT4_MAP_UNWRITTEN flag. But it should be fixed.
--- fs/ext4/extents.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/ext4/extents.c +4 -0
@@ -3626,6 +3626,10 @@ ext4_ext_handle_uninitialized_extents(handle_t *handle, struct inode *inode, path, map->m_len); }else err = ret; + map->m_flags |=EXT4_MAP_MAPPED ; +if (allocated > map->m_len) + allocated = map->m_len; + map->m_len = allocated; goto out2; }/* buffered IO case */
Signature · 1 line
1.7.12.rc2.18.g61b472e
Re: [PATCH] ext4: fix wrong m_len value after unwritten extent conversion (Re: [PATCH 5/5] ext4: invalidate...)
Quoted 108 lines · Zheng Liu
Hi Dmitry and Ted,
On Mon, Feb 25, 2013 at 08:29:46PM +0400, Dmitry Monakhov wrote:
[snip]Quoted 53 lines
One BIG note about this patch.
It fixBUG_ON(bh->b_blocknr != pblock)infs/ext4/inode.c:1452but 300'th xfstest(from my repo) still failed due to data corruption in verifier thread.
LOG:MOUNT_OPTIONS-- -o acl,user_xattr/dev/mapper/vzvg-scratch_dev/mnt_scratch300 33s ... [failed, exit status 1] - output mismatch (see 300.out.bad)
--- 300.out 2013-02-20 12:46:24.000000000 +0400 +++ 300.out.bad 2013-02-25 20:18:24.000000000 +0400 @@ -2,3 +2,5 @@ Start defragment activity +failed: '/usr/local/bin/fio /tmp/1665-300.fio' +(see 300.full for details) ... (Run 'diff -u 300.out 300.out.bad' to see the entire diff)#300.full
crc32c: verify failed at file /mnt_scratch/test2 offset 16973824, length65536 Expected CRC: d062565e Received CRC: f2cd2028 received data dumped as test2.16973824.received expected data dumped as test2.16973824.expecteddefrag-4k: Laying out IO file(s) (1 file(s) / 3400MB)
donor-file-fuzzer: Laying out IO file(s) (1 file(s) / 3400MB)#DIFF:
---/tmp/test2.16973824.expected2013-02-25 20:23:04.000000000+0400 +++ /tmp/test2.16973824.received 2013-02-25 20:22:55.000000000 +0400(quoted diff) +2
-9@@ -254,3844 +254,6 @@ 0000fd0 eb95 6201 89ec 151a bd72 9675 203c 0d80 0000fe0 b7ae b415 b8cc 0b2f b6f5 baab 9d0e 1741 0000ff0 f6de fbda d64b 1bd3 5edb 040c 7cdc 18e6 -0001000 0bdb 5114 cd95 01eb 017b a435 d128 11af -0001010 202f 93c9 a80a 013e a405 c261 8b1c 0dab -0001020 b480 c649 1032 1b0a 3690 7e89 dee1 0ac7 -0001030 26d2 9489 3c97 0bd8 24da 5f28 3d4e 066d -0001040 049b b978 7815 159c 8093 b4e1 b246 1c25 -000ffc0 bbb1 fa68 5622 022d 9776 0174 2d90 1ecb -000ffd0 92ee b473 64f7 0bcb 725d 2d17 9265 0fff -000ffe0 6e4b ec74 5bc0 0618 0dc9 e669 953d 002f -000fff0 a1b9 35e8 ff73 17a5 9437 15e0 24fa 150a +0001000 0000 0000 0000 0000 0000 0000 0000 0000 +* 0010000Quoted 3 lines
It seems that we data was written to uninitialized extent, but
unwritten->initializedextent conversion was missed somewhere. I have not fix for that issue yet.I take a close look at this bug, and I found that a wrong '
map->m_len' is returned fromext4_ext_map_blockswhen we try to convert an unwritten extent withEXT4_GET_BLOCKS_IO_CONVERT_EXTflag. Here we always assume that the return value of ext/ind_map_blocks() is equal tom_len. But here it breaks this assumption. Let us consider what happens.When we do a dio write for a extent-based file, we will preallocate an unwritten extent for it. After dio write has been done, we will convert it in
ext4_end_iocallback. We useext4_convert_unwritten_extentsto do this conversion. Then the codepath is like beblow:
ext4_convert_unwritten_extents()
->ext4_map_blocks()->ext4_es_lookup_extent() [We can lookup an unwritten extent] ->ext4_ext_map_blocks() ->ext4_ext_handle_uninitialized_extents() ->ext4_es_insert_extent() [We update status tree, but the length of written extent is wrong. The length of written extent is greater than the length of we have converted]The following case demonstrates what happens.
1. We do a dio write. An unwritten extent will be preallocated.
status tree: [0, 23]:unwritten
extent tree: [0, 23]:unwritten2. We try to convert an unwritten extent, e.g. [0, 15]. But due to this bug we will update all unwritten extent in status tree.
status tree: [0, 23]:written
extent tree: [0, 15]:written, [16, 23]:unwritten3. Then we try to convert the following unwritten extent, but we will return directly in
ext4_map_blocksbecause we lookup an written extent from status tree, and that unwritten extent will never be converted. At the time if e4defrag is running, the status tree could be removed, and we could read an unwritten extent.Certainly this is only my *guess* because in my sandbox xfstests #300 and #301 never fail. I am not sure this patch could fix this regression. *But* I am pretty sure it will cause some potential bugs if it hasn't been fixed. I paste the patch at the below, which has been tested by xfstests plus Dmitry's #300 and #301 test case at the following configurations.
Quoted 8 lines
* Ext4 4k block
* Ext4 4k block w/dioread_nolock
* EXt4 4k block w/bigallocDmitry, I really appreciate if you could give this patch a try. Hope it can fix this regression. Please let me know if there is any update. Thank you very much.
No 300'th still fails. Even more it trigger new error:
EXT4-fs error (device dm-3): ext4_move_extents:1486: inode #12: comm
fio: We replaced blocks too much! sum of replaced: 33 requested: 32
BTW I dont understand your patch.
You state that (map->m_len > ex->ee_len ) condition is possible inside ext4_convert_unwritten_extents_endio() but HOW?
0) We call dio [lblk:10, len:10] to fallocated area [lblk:0, len:30, unwritten]
1)DIO preparation call map_blocks with EXT4_GET_BLOCKS_PRE_IO
->ext4_ext_handle_uninitialized_extents
->ext4_split_unwritten_extents
which split extent according to map->m_len -> [0,10,u], [10,10,u], [20,10,u]2) ext_end_io will call map_blocks with EXT4_GET_BLOCKS_CONVERT ->ext4_ext_handle_uninitialized_extents
->ext4_convert_unwritten_extents_endio
will found [10,10,u] and convert it to [10,10,w]e4defrag can not change layout between (1)-(2) because it properly wait for any outstanding aio like follows (move_extent.c:1328):
/* Protect orig and donor inodes against a truncate */
mext_inode_double_lock(orig_inode, donor_inode);
/* Wait for all existing dio workers */
ext4_inode_block_unlocked_dio(orig_inode);
ext4_inode_block_unlocked_dio(donor_inode);
inode_dio_wait(orig_inode);
inode_dio_wait(donor_inode)
/* Protect extent tree against block allocations via delalloc */
double_down_write_data_sem(orig_inode, donor_inode);
/*Here any io activity is blocked for given inodes. */Quoted 48 lines
Regards,
- ZhengSubject: [PATCH] ext4: fix wrong
m_lenvalue after unwritten extent conversionFrom: Zheng Liu <[email protected]>
We always assume that the return value of
ext4_ext_map_blocksis equal tomap->m_lenbut when we try to convert an unwritten extent 'm_len' value will break this assumption. It is harmless until we use status tree as a extent cache because we need to update status tree according to 'm_len' value.Meanwhile this commit marks
EXT4_MAP_MAPPEDflag after unwritten extent conversion. It shouldn't cause a bug because we update status tree according to checkingEXT4_MAP_UNWRITTENflag. But it should be fixed.Signed-off-by Zheng Liu <[email protected]>Cc "Theodore Ts'o" <[email protected]>Cc Dmitry Monakhov <[email protected]>--- fs/ext4/extents.c | 4 ++++ 1 file changed, 4 insertions(+)fs/ext4/extents.c +4
-0@@ -3626,6 +3626,10 @@ ext4_ext_handle_uninitialized_extents(handle_t *handle, struct inode *inode, path, map->m_len); }else err = ret; + map->m_flags |=EXT4_MAP_MAPPED ; +if (allocated > map->m_len) + allocated = map->m_len; + map->m_len = allocated; goto out2; }/* buffered IO case */ Signature · 6 lines
1.7.12.rc2.18.g61b472e -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 5/5] ext4: invalidate exntent-status-tree during extent_migration
In reply to Dmitry MonakhovQuoted 7 lines · Dmitry Monakhov
mext_replace_branches()will change inode's extents layout so we have to drop corresponding cache.TESTCASE: 301'th xfstest was not yet accepted to official xfstest's branch
and can be found here: https://github.com/dmonakhov/xfstests/commit/7b7efeee30a41109201e2040034e71db9b66ddc0Signed-off-by Dmitry Monakhov <[email protected]>
Makes sense. You can add:
Honza
Quoted 26 lines
--- fs/ext4/move_extent.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)fs/ext4/move_extent.c +8
-0@@ -666,6 +666,14 @@ mext_replace_branches(handle_t *handle, struct inode *orig_inode,int replaced_count =0 ;int dext_alen; + *err =ext4_es_remove_extent (orig_inode, from, count); +if (*err) + goto out; + + *err =ext4_es_remove_extent (donor_inode, from, count); +if (*err) + goto out; +/* Get the original extent for the block "orig_off" */ *err =get_ext_path (orig_inode, orig_off, &orig_path);if (*err)Signature · 1 line
1.7.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 17 lines · Dmitry Monakhov
When
ext4_split_extent_at()ends up doing zeroout & conversion to initialized instead of split & conversion,ext4_split_extent()gets confused and can wrongly mark the extent back as uninitialized resulting in end IO code getting confused from large unwritten extents and may result in data loss.The example of problematic behavior is:
lblk len lblk len
ext4_split_extent()(ex=[1000,30,uninit], map=[1010,10])ext4_split_extent_at() (split [1000,30,uninit] at 1020) ext4_ext_insert_extent() -> ENOSPC ext4_ext_zeroout() -> extent [1000,30] is now initialized ext4_split_extent_at() (split [1000,30,init] at 1010, MARK_UNINIT1 | MARK_UNINIT2) -> extent is split and parts marked as uninitialized
Quoted 8 lines
Fix the problem by rechecking extent type after the first
ext4_split_extent_at()returns. None ofsplit_flagscan not be applied to initialized extent so this patch also addBUG_ONto prevent similar issues in future.TESTCASE: https://github.com/dmonakhov/xfstests/commit/b8a55eb5ce28c6ff29e620ab090902fcd5833597
Changes since V2: Patch no longer depends on Jan's "disable-uninit-ext-mergring" patch.
Looks good. You can add:
Honza
Quoted 56 lines
Signed-off-by Dmitry Monakhov <[email protected]>--- fs/ext4/extents.c | 22 ++++++++++++++++------ 1 files changed, 16 insertions(+), 6 deletions(-)fs/ext4/extents.c +16
-6@@ -2943,6 +2943,10 @@ static int ext4_split_extent_at(handle_t *handle, newblock = split - ee_block +ext4_ext_pblock (ex);BUG_ON (split < ee_block || split >= (ee_block + ee_len)); +BUG_ON (!ext4_ext_is_uninitialized (ex) && + split_flag & (EXT4_EXT_MAY_ZEROOUT | +EXT4_EXT_MARK_UNINIT1 | +EXT4_EXT_MARK_UNINIT2 )); err =ext4_ext_get_access (handle, inode, path + depth);if (err)@@ -3061,19 +3065,25 @@ static int ext4_split_extent(handle_t *handle,if (err) goto out; } - +/* +* Update path is required because previous ext4_split_extent_at() may +* result in split of original leaf or extent zeroout. +*/ ext4_ext_drop_refs (path); path =ext4_ext_find_extent (inode, map->m_lblk, path);if (IS_ERR (path))return PTR_ERR (path); + depth =ext_depth (inode); + ex = path[depth].p_ext; + uninitialized =ext4_ext_is_uninitialized (ex); + split_flag1 =0 ;if (map->m_lblk >= ee_block) { - split_flag1 = split_flag & (EXT4_EXT_MAY_ZEROOUT | -EXT4_EXT_DATA_VALID2 ); -if (uninitialized) + split_flag1 = split_flag &EXT4_EXT_DATA_VALID2 ; +if (uninitialized) { split_flag1 |=EXT4_EXT_MARK_UNINIT1 ; -if (split_flag &EXT4_EXT_MARK_UNINIT2 ) - split_flag1 |=EXT4_EXT_MARK_UNINIT2 ; + split_flag1 |= split_flag & (EXT4_EXT_MAY_ZEROOUT |EXT4_EXT_MARK_UNINIT2 ); + } err =ext4_split_extent_at (handle, inode, path, map->m_lblk, split_flag1, flags);if (err)Signature · 1 line
1.7.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Theodore Ts'o
maintainerDmitry,
Thanks for working on these patches. They look good! I've dropped them into the ext4 tree in the dev branch, and am starting to run tests on them.
Zheng, since some of your fix up patches look like they touch some of the code modified by Dmitry's patches, could you rebase your patch set on top of the dev branch (which is what we've pushed to Linus plus
Dmitry's patches)? Thanks!!
- Ted
Quoted 10 lines · Theodore Ts'o
Dmitry,
Thanks for working on these patches. They look good! I've dropped them into the ext4 tree in the dev branch, and am starting to run tests on them.
Zheng, since some of your fix up patches look like they touch some of the code modified by Dmitry's patches, could you rebase your patch set on top of the dev branch (which is what we've pushed to Linus plus
Dmitry's patches)? Thanks!!
No problem, I will rebase my patches against the latest dev branch.
Thanks,
- Zheng