ext4: stop return ENOSPC from ext4_issue_zeroout
PATCH 1/1 ext4: stop return ENOSPC from ext4_issue_zeroout
Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since ext4_writepages fall into a infinite loop:
static int ext4_writepages(xxx)
{
...
while (!done && mpd.first_page <= mpd.last_page) {
...
ret = mpage_prepare_extent_to_map(&mpd);
if (!ret) {
...
ret = mpage_map_and_submit_extent(handle,
&mpd,&give_up_on_write);
<----- will return -ENOSPC
...
}
...
if (ret == -ENOSPC && sbi->s_journal) {
<------ we cannot break since we will get ENOSPC forever
jbd2_journal_force_commit_nested(sbi->s_journal);
ret = 0;
continue;
}
... }
}
Got ENOSPC with follow stack:
...ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC
'df22291ff0fd ("ext4: Retry block allocation if we have free blocks left")' add the logic to retry block allcate since we may get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead the upper infinite loop. Fix it by convert the err to EIO.
--- fs/ext4/inode.c | 3 +++ 1 file changed, 3 insertions(+)
fs/ext4/inode.c +3 -0
@@ -428,6 +428,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,if (ret >0 ) ret =0 ; +if (ret == -ENOSPC ) + ret = -EIO ; +return ret; }
Signature · 1 line
2.31.1
Quoted 37 lines · yangerkun
Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since
ext4_writepagesfall into a infinite loop:static int
ext4_writepages(xxx){ ... while (!done && mpd.first_page <= mpd.last_page) { ... ret = mpage_prepare_extent_to_map(&mpd); if (!ret) { ... ret = mpage_map_and_submit_extent(handle, &mpd,&give_up_on_write); <----- will return -ENOSPC ... } ... if (ret == -ENOSPC && sbi->s_journal) { <------ we cannot break since we will get ENOSPC forever jbd2_journal_force_commit_nested(sbi->s_journal); ret = 0; continue; } ...}
}Got ENOSPC with follow stack:
...ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait<-- bio to thinpool will return ENOSPC
Quoted 7 lines
'
df22291ff0fd("ext4: Retry block allocation if we have free blocks left")' add the logic to retry block allcate since we may get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead the upper infinite loop. Fix it by convert the err to EIO.Signed-off-by yangerkun <[email protected]>
Thanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:
Honza
Quoted 21 lines
--- fs/ext4/inode.c | 3 +++ 1 file changed, 3 insertions(+)fs/ext4/inode.c +3
-0@@ -428,6 +428,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,if (ret >0 ) ret =0 ; +if (ret == -ENOSPC ) + ret = -EIO ; +return ret; }Signature · 1 line
2.31.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Theodore Ts'o
maintainerQuoted 15 lines · Jan Kara
Quoted 4 lines · yangerkun
Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since
ext4_writepagesfall into a infinite loop:Quoted 9 lines
Got ENOSPC with follow stack:
...ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait<-- bio to thinpool will return ENOSPC
Quoted 7 lines
Thanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:
Reviewed-by Jan Kara <[email protected]>
Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?
- Ted
fs/ext4/extents.c +3 -2
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Theodore Ts'o
maintainerPATCH 1/1 ext4: if zeroout fails fall back to splitting the extent node
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
---
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
fs/ext4/extents.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Signature · 1 line
2.31.0
Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 12 lines · Theodore Ts'o
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit
df22291ff0fd("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
Quoted 7 lines
Reported-by yangerkun <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>---
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
Will do it.
Quoted 35 lines
fs/ext4/extents.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/ext4/extents.c +3
-2@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 19 lines · yangerkun
Quoted 16 lines · Theodore Ts'o
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit
df22291ff0fd("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
Reported-by yangerkun <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>---
Quoted 5 lines
Quoted 3 lines
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
Will do it.
Thanks for the patch, it can help us to pass the testcase. And after some review, it's really a better fix for me.
Thanks,
Kun.
Quoted 38 lines
Quoted 30 lines
fs/ext4/extents.c| 5 +++--1 file changed, 3 insertions(+), 2 deletions(-)fs/ext4/extents.c +1
-1@@ -3569,7 +3569,7 @@ static intext4_ext_convert_to_initialized (handle_t *handle, split_map.m_len - ee_block); err = ext4_ext_zeroout(inode, &zero_ex1); if (err) - goto out; + goto fallback; split_map.m_len = allocated; }if (split_map.m_lblk -
ee_block+ split_map.m_len <fs/ext4/extents.c +1
-1@@ -3583,7 +3583,7 @@ static intext4_ext_convert_to_initialized (handle_t *handle, ext4_ext_pblock(ex)); err = ext4_ext_zeroout(inode, &zero_ex2); if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;fs/ext4/extents.c static int · line 3598
Quoted 7 lines
fs/ext4/extents.c +1
-0@@ -3592,6 +3592,7 @@ static intext4_ext_convert_to_initialized (handle_t *handle, } } +fallback: err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag, flags); if (err > 0).
Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
In reply to Theodore Ts'oHi,
Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.
fs/ext4/extents.c +2 -0
@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle, goto out;if (EXT4_EXT_MAY_ZEROOUT & split_flag) { +int ret =0 ; + if (split_flag &
(EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
if (split_flag & EXT4_EXT_DATA_VALID1) {
- err = ext4_ext_zeroout(inode, ex2);
+ ret = ext4_ext_zeroout(inode, ex2);
zero_ex.ee_block = ex2->ee_block;
zero_ex.ee_len = cpu_to_le16(ext4_ext_get_actual_len(ex2));
ext4_ext_store_pblock(&zero_ex,ext4_ext_pblock(ex2));
} else {
- err = ext4_ext_zeroout(inode, ex);
+ ret = ext4_ext_zeroout(inode, ex);
zero_ex.ee_block = ex->ee_block;
zero_ex.ee_len = cpu_to_le16(ext4_ext_get_actual_len(ex));
@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle,
ext4_ext_pblock(ex));
}
} else {
- err = ext4_ext_zeroout(inode, &orig_ex);
+ ret = ext4_ext_zeroout(inode, &orig_ex);
zero_ex.ee_block = orig_ex.ee_block;
zero_ex.ee_len = cpu_to_le16(ext4_ext_get_actual_len(&orig_ex));
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle,
ext4_ext_pblock(&orig_ex));
}- if (!err) {
+ if (!ret) {
/* update the extent length and mark as
initialized */
ex->ee_len = cpu_to_le16(ee_len);
ext4_ext_try_to_merge(handle, inode, path, ex);fs/ext4/inode.c +3 -0
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,if (ret >0 ) ret =0 ; +if (ret == -ENOSPC ) + ret = -EIO ; +return ret; }
Quoted 54 lines · Theodore Ts'o
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit
df22291ff0fd("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
Reported-by yangerkun <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>---
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
fs/ext4/extents.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/ext4/extents.c +3
-2@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Jan Kara
reviewerRe: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Hello,
Quoted 10 lines · yangerkun
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problemtoo(in theory, not really happened)...How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by
ext4_ext_insert_extentinext4_split_extent_at.Besides,
308c57ccf431("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?
Honza
Quoted 119 lines
fs/ext4/extents.c +4
-2@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle, goto out;if (EXT4_EXT_MAY_ZEROOUT & split_flag) { +int ret =0 ; +if (split_flag & (EXT4_EXT_DATA_VALID1 |EXT4_EXT_DATA_VALID2 )) {if (split_flag &EXT4_EXT_DATA_VALID1 ) { - err =ext4_ext_zeroout (inode, ex2); + ret =ext4_ext_zeroout (inode, ex2); zero_ex.ee_block = ex2->ee_block; zero_ex.ee_len =cpu_to_le16 (ext4_ext_get_actual_len (ex2));ext4_ext_store_pblock (&zero_ex,ext4_ext_pblock (ex2)); }else { - err =ext4_ext_zeroout (inode, ex); + ret =ext4_ext_zeroout (inode, ex);zero_ex.ee_block = ex->ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(ex));fs/ext4/extents.c +1
-1@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(ex)); } }else { - err =ext4_ext_zeroout (inode, &orig_ex); + ret =ext4_ext_zeroout (inode, &orig_ex); zero_ex.ee_block = orig_ex.ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(&orig_ex));fs/ext4/extents.c +1
-1@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); } -if (!err) { +if (!ret) {/* update the extent length and mark as initialized */ ex->ee_len =cpu_to_le16 (ee_len);ext4_ext_try_to_merge(handle, inode, path, ex);fs/ext4/inode.c +3
-0@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk, if (ret > 0) ret = 0; +if (ret == -ENOSPC ) + ret = -EIO ; +return ret; }Quoted 52 lines · Theodore Ts'o
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit
df22291ff0fd("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
Reported-by yangerkun <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>---
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
fs/ext4/extents.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/ext4/extents.c +3
-2@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 14 lines · Jan Kara
Hello,
Quoted 10 lines · yangerkun
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problemtoo(in theory, not really happened)...How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by
ext4_ext_insert_extentinext4_split_extent_at.Besides,
308c57ccf431("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.
Quoted 6 lines
I've got back to this. The
ext4_ext_zeroout()calls inext4_split_extent_at()seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If evenext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?
I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...
Quoted 122 lines
Honza
Quoted 119 lines
fs/ext4/extents.c +4
-2@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle, goto out;if (EXT4_EXT_MAY_ZEROOUT & split_flag) { +int ret =0 ; +if (split_flag & (EXT4_EXT_DATA_VALID1 |EXT4_EXT_DATA_VALID2 )) {if (split_flag &EXT4_EXT_DATA_VALID1 ) { - err =ext4_ext_zeroout (inode, ex2); + ret =ext4_ext_zeroout (inode, ex2); zero_ex.ee_block = ex2->ee_block; zero_ex.ee_len =cpu_to_le16 (ext4_ext_get_actual_len (ex2));ext4_ext_store_pblock (&zero_ex,ext4_ext_pblock (ex2)); }else { - err =ext4_ext_zeroout (inode, ex); + ret =ext4_ext_zeroout (inode, ex);zero_ex.ee_block = ex->ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(ex));fs/ext4/extents.c +1
-1@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(ex)); } }else { - err =ext4_ext_zeroout (inode, &orig_ex); + ret =ext4_ext_zeroout (inode, &orig_ex); zero_ex.ee_block = orig_ex.ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(&orig_ex));fs/ext4/extents.c +1
-1@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); } -if (!err) { +if (!ret) {/* update the extent length and mark as initialized */ ex->ee_len =cpu_to_le16 (ee_len);ext4_ext_try_to_merge(handle, inode, path, ex);fs/ext4/inode.c +3
-0@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk, if (ret > 0) ret = 0; +if (ret == -ENOSPC ) + ret = -EIO ; +return ret; }Quoted 52 lines · Theodore Ts'o
If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.
Commit
df22291ff0fd("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.
Reported-by yangerkun <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>---
I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?
fs/ext4/extents.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/ext4/extents.c +3
-2@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Jan Kara
reviewerRe: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara
Hello,
Quoted 10 lines · yangerkun
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problemtoo(in theory, not really happened)...How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by
ext4_ext_insert_extentinext4_split_extent_at.Besides,
308c57ccf431("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.Quoted 6 lines
I've got back to this. The
ext4_ext_zeroout()calls inext4_split_extent_at()seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If evenext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?
Quoted 3 lines
I am afraid about the EDQUOT from
ext4_ext_insert_extentmay be overwrite byext4_ext_zerooutwith ENOSPC. And this may lead to dead loop sinceext4_writepageswill retry once get ENOSPC? Maybe I am wrong...
OK, so passing back original error instead of the error from ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at() called from ext4_writepages() because we should have reserved enough space for extent splits when writing data. So hitting that is already unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug. Secondly, returning EIO instead of ENOSPC is IMO a bit confusing for upper layers and makes it harder to analyze where the real problem is...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 27 lines · Jan Kara
Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara
Hello,
Quoted 10 lines · yangerkun
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problemtoo(in theory, not really happened)...How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by
ext4_ext_insert_extentinext4_split_extent_at.Besides,
308c57ccf431("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.Quoted 6 lines
I've got back to this. The
ext4_ext_zeroout()calls inext4_split_extent_at()seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If evenext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?Quoted 3 lines
I am afraid about the EDQUOT from
ext4_ext_insert_extentmay be overwrite byext4_ext_zerooutwith ENOSPC. And this may lead to dead loop sinceext4_writepageswill retry once get ENOSPC? Maybe I am wrong...
Quoted 5 lines
OK, so passing back original error instead of the error from
ext4_ext_zeroout()makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen inext4_split_extent_at()called fromext4_writepages()because we should have reserved enough space for extent splits when writing data. So hitting that is already
ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_space
It seems we will only reserve space for data, no for metadata...
Quoted 3 lines
unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug.
Agree. EIO from ext4_ext_zeroout that overwrite the ENOSPC from ext4_ext_insert_extent seems buggy too. Maybe we should ignore the error from ext4_ext_zeroout and return the error from ext4_ext_insert_extent once ext4_ext_zeroout in ext4_split_extent_at got a error. Something like this:
fs/ext4/extents.c +1 -0
@@ -3185,6 +3185,7 @@ static int ext4_split_extent_at(handle_t *handle,struct ext4_extent *ex2 =NULL ;unsigned int ee_len, depth;int err =0 ; +int err1; BUG_ON((split_flag & (EXT4_EXT_DATA_VALID1 |
EXT4_EXT_DATA_VALID2)) ==
(EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2));
@@ -3255,7 +3256,7 @@ static int ext4_split_extent_at(handle_t *handle,
if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
if (split_flag &
(EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
if (split_flag & EXT4_EXT_DATA_VALID1) {
- err = ext4_ext_zeroout(inode, ex2);
+ err1 = ext4_ext_zeroout(inode, ex2);
zero_ex.ee_block = ex2->ee_block;
zero_ex.ee_len = cpu_to_le16(ext4_ext_get_actual_len(ex2));
@@ -3270,7 +3271,7 @@ static int ext4_split_extent_at(handle_t *handle,
ext4_ext_pblock(ex));
}
} else {
- err = ext4_ext_zeroout(inode, &orig_ex);
+ err1 = ext4_ext_zeroout(inode, &orig_ex);
zero_ex.ee_block = orig_ex.ee_block;
zero_ex.ee_len = cpu_to_le16(ext4_ext_get_actual_len(&orig_ex));
@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle,
ext4_ext_pblock(&orig_ex));
}- if (!err) {
+ if (!err1) {
/* update the extent length and mark as
initialized */
ex->ee_len = cpu_to_le16(ee_len);
ext4_ext_try_to_merge(handle, inode, path, ex);Quoted 6 lines
Secondly, returning EIO instead of ENOSPC is IMO a
bit confusing for upper layers and makes it harder to analyze where the real problem is...Honza
Jan Kara
reviewerRe: [PATCH] ext4: if zeroout fails fall back to splitting the extent node
Quoted 48 lines · yangerkun
Quoted 27 lines · Jan Kara
Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara
Hello,
Quoted 10 lines · yangerkun
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problemtoo(in theory, not really happened)...How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by
ext4_ext_insert_extentinext4_split_extent_at.Besides,
308c57ccf431("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.Quoted 6 lines
I've got back to this. The
ext4_ext_zeroout()calls inext4_split_extent_at()seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If evenext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?Quoted 3 lines
I am afraid about the EDQUOT from
ext4_ext_insert_extentmay be overwrite byext4_ext_zerooutwith ENOSPC. And this may lead to dead loop sinceext4_writepageswill retry once get ENOSPC? Maybe I am wrong...Quoted 5 lines
OK, so passing back original error instead of the error from
ext4_ext_zeroout()makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen inext4_split_extent_at()called fromext4_writepages()because we should have reserved enough space for extent splits when writing data. So hitting that is already
ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_spaceIt seems we will only reserve space for data, no for metadata...
Quoted 3 lines
unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug.
Quoted 5 lines
Agree. EIO from
ext4_ext_zerooutthat overwrite the ENOSPC fromext4_ext_insert_extentseems buggy too. Maybe we should ignore the error fromext4_ext_zerooutand return the error fromext4_ext_insert_extentonceext4_ext_zerooutinext4_split_extent_atgot a error. Something like this:
Yep, something like that looks good to me.
Honza
Quoted 54 lines
fs/ext4/extents.c +1
-0@@ -3185,6 +3185,7 @@ static int ext4_split_extent_at(handle_t *handle,struct ext4_extent *ex2 =NULL ;unsigned int ee_len, depth;int err =0 ; +int err1; BUG_ON((split_flag & (EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2))==
(EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2));fs/ext4/extents.c +1
-1@@ -3255,7 +3256,7 @@ static int ext4_split_extent_at(handle_t *handle,if (EXT4_EXT_MAY_ZEROOUT & split_flag) {if (split_flag & (EXT4_EXT_DATA_VALID1 |EXT4_EXT_DATA_VALID2 )) {if (split_flag &EXT4_EXT_DATA_VALID1 ) { - err =ext4_ext_zeroout (inode, ex2); + err1 =ext4_ext_zeroout (inode, ex2); zero_ex.ee_block = ex2->ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(ex2));fs/ext4/extents.c +1
-1@@ -3270,7 +3271,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(ex)); } }else { - err =ext4_ext_zeroout (inode, &orig_ex); + err1 =ext4_ext_zeroout (inode, &orig_ex); zero_ex.ee_block = orig_ex.ee_block; zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(&orig_ex));fs/ext4/extents.c +1
-1@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); } -if (!err) { +if (!err1) {/* update the extent length and mark as initialized */ ex->ee_len =cpu_to_le16 (ee_len);ext4_ext_try_to_merge(handle, inode, path, ex);Quoted 6 lines
Secondly, returning EIO instead of ENOSPC is IMO a
bit confusing for upper layers and makes it harder to analyze where the real problem is...Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
reviewerQuoted 22 lines · Theodore Ts'o
Quoted 21 lines · Jan Kara
Quoted 4 lines · yangerkun
Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since
ext4_writepagesfall into a infinite loop:Quoted 9 lines
Got ENOSPC with follow stack:
...ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait<-- bio to thinpool will return ENOSPCThanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:
Quoted 4 lines
Quoted 1 line
Reviewed-by Jan Kara <[email protected]>Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?
Yes, that looks indeed better. I'd note that even splitting extent may fail due to ENOSPC on thin-provisioned storage but the chances are *much* lower.
Honza
Quoted 30 lines
fs/ext4/extents.c +3
-2@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, split_map.m_len - ee_block); err =ext4_ext_zeroout (inode, &zero_ex1);if (err) - goto out; + goto fallback; split_map.m_len = allocated; } if (split_map.m_lblk - ee_block + split_map.m_len <@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, ext4_ext_pblock(ex)); err =ext4_ext_zeroout (inode, &zero_ex2);if (err) - goto out; + goto fallback; } split_map.m_len += split_map.m_lblk - ee_block;@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle, } } +fallback: err =ext4_split_extent (handle, inode, ppath, &split_map, split_flag, flags);if (err >0 )
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Theodore Ts'o
maintainerQuoted 3 lines · Jan Kara
Yes, that looks indeed better. I'd note that even splitting extent may fail due to ENOSPC on thin-provisioned storage but the chances are *much* lower.
Indeed, any kind of metadata update (updating an inode atime, creating a new inode, deleting a directory entry, etc.) can fail due to ENOSPC on thin-provision storage, leading to a potentially corrupted file system since some writes will succeed, while others won't --- and that's not a scenario that's super well tested, nor is there much we can do other than remounting the file system read/only or forcing a reboot.
But at least we won't throw the kernel into an infinite loop, which is what yangerkun was reporting...
- Ted