Skip to contentSkip to conversation

ext4: stop return ENOSPC from ext4_issue_zeroout

PATCH In review yangerkun 3 participants 14 messages Aug 4, 2021 – Nov 24, 2021 ext4 file system

lore
Share & tools
lore

yangerkun

author
#0 lore raw

PATCH 1/1 ext4: stop return ENOSPC from ext4_issue_zeroout

Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since ext4_writepages fall into a infinite loop:

static int ext4_writepages(xxx)

{
    ...
   while (!done && mpd.first_page <= mpd.last_page) {
       ...
       ret = mpage_prepare_extent_to_map(&mpd);
       if (!ret) {
           ...
           ret = mpage_map_and_submit_extent(handle,
&mpd,&give_up_on_write);
           <----- will return -ENOSPC
           ...
       }
       ...
       if (ret == -ENOSPC && sbi->s_journal) {
           <------ we cannot break since we will get ENOSPC forever
           jbd2_journal_force_commit_nested(sbi->s_journal);
           ret = 0;
           continue;
       }
       ...

}
}

Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC

'df22291ff0fd ("ext4: Retry block allocation if we have free blocks left")' add the logic to retry block allcate since we may get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead the upper infinite loop. Fix it by convert the err to EIO.

---
 fs/ext4/inode.c | 3 +++
 1 file changed, 3 insertions(+)
fs/ext4/inode.c +3 -0
index 038aebd7eb2f..d9ded699a88c 100644
@@ -428,6 +428,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,
 	if (ret > 0)
 		ret = 0;
 
+	if (ret == -ENOSPC)
+		ret = -EIO;
+
 	return ret;
 }
 
Signature · 1 line
2.31.1
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 37 lines · yangerkun

Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since ext4_writepages fall into a infinite loop:

static int ext4_writepages(xxx)

{
    ...
   while (!done && mpd.first_page <= mpd.last_page) {
       ...
       ret = mpage_prepare_extent_to_map(&mpd);
       if (!ret) {
           ...
           ret = mpage_map_and_submit_extent(handle,
&mpd,&give_up_on_write);
           <----- will return -ENOSPC
           ...
       }
       ...
       if (ret == -ENOSPC && sbi->s_journal) {
           <------ we cannot break since we will get ENOSPC forever
           jbd2_journal_force_commit_nested(sbi->s_journal);
           ret = 0;
           continue;
       }
       ...

}
}

Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC

Quoted 7 lines

'df22291ff0fd ("ext4: Retry block allocation if we have free blocks left")' add the logic to retry block allcate since we may get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead the upper infinite loop. Fix it by convert the err to EIO.

Thanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:

Honza

Quoted 21 lines
---
 fs/ext4/inode.c | 3 +++
 1 file changed, 3 insertions(+)
fs/ext4/inode.c +3 -0
index 038aebd7eb2f..d9ded699a88c 100644
@@ -428,6 +428,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,
 	if (ret > 0)
 		ret = 0;
 
+	if (ret == -ENOSPC)
+		ret = -EIO;
+
 	return ret;
 }
 
Signature · 1 line
2.31.1
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#2 lore raw
Quoted 15 lines · Jan Kara
Quoted 4 lines · yangerkun

Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since ext4_writepages fall into a infinite loop:

Quoted 9 lines

Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC

Quoted 7 lines

Thanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:

Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?

- Ted

fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 				split_map.m_len - ee_block);
 			err = ext4_ext_zeroout(inode, &zero_ex1);
 			if (err)
-				goto out;
+				goto fallback;
 			split_map.m_len = allocated;
 		}
 		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 						      ext4_ext_pblock(ex));
 				err = ext4_ext_zeroout(inode, &zero_ex2);
 				if (err)
-					goto out;
+					goto fallback;
 			}
 
 			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 		}
 	}
 
+fallback:
 	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
 				flags);
 	if (err > 0)
maintainer
#3 lore raw

PATCH 1/1 ext4: if zeroout fails fall back to splitting the extent node

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

---

I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?

 fs/ext4/extents.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 				split_map.m_len - ee_block);
 			err = ext4_ext_zeroout(inode, &zero_ex1);
 			if (err)
-				goto out;
+				goto fallback;
 			split_map.m_len = allocated;
 		}
 		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 						      ext4_ext_pblock(ex));
 				err = ext4_ext_zeroout(inode, &zero_ex2);
 				if (err)
-					goto out;
+					goto fallback;
 			}
 
 			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 		}
 	}
 
+fallback:
 	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
 				flags);
 	if (err > 0)
Signature · 1 line
2.31.0
author
#4 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 12 lines · Theodore Ts'o

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

Quoted 7 lines

---

I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?

Will do it.

Quoted 35 lines
  fs/ext4/extents.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  				split_map.m_len - ee_block);
  			err = ext4_ext_zeroout(inode, &zero_ex1);
  			if (err)
-				goto out;
+				goto fallback;
  			split_map.m_len = allocated;
  		}
  		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  						      ext4_ext_pblock(ex));
  				err = ext4_ext_zeroout(inode, &zero_ex2);
  				if (err)
-					goto out;
+					goto fallback;
  			}
 
  			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  		}
  	}
 
+fallback:
  	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
  				flags);
  	if (err > 0)
author
#5 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 19 lines · yangerkun
Quoted 16 lines · Theodore Ts'o

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

---

Quoted 5 lines
Quoted 3 lines

I've run this through my battery of tests, and it doesn't cause any regressions.  Yangerkun, can you test this and see if this works for you?

Will do it.

Thanks for the patch, it can help us to pass the testcase. And after some review, it's really a better fix for me.

Thanks,
Kun.

Quoted 38 lines
Quoted 30 lines

  fs/ext4/extents.c | 5 +++--

  1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +1 -1
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int 
 ext4_ext_convert_to_initialized(handle_t *handle,
                   split_map.m_len - ee_block);
               err = ext4_ext_zeroout(inode, &zero_ex1);
               if (err)
-                goto out;
+                goto fallback;
               split_map.m_len = allocated;
           }

          if (split_map.m_lblk - ee_block + split_map.m_len <

fs/ext4/extents.c +1 -1
@@ -3583,7 +3583,7 @@ static int 
 ext4_ext_convert_to_initialized(handle_t *handle,
                                 ext4_ext_pblock(ex));
                   err = ext4_ext_zeroout(inode, &zero_ex2);
                   if (err)
-                    goto out;
+                    goto fallback;
               }
               split_map.m_len += split_map.m_lblk - ee_block;

fs/ext4/extents.c static int · line 3598

Quoted 7 lines
fs/ext4/extents.c +1 -0
@@ -3592,6 +3592,7 @@ static int 
 ext4_ext_convert_to_initialized(handle_t *handle,
           }
       }
+fallback:
       err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
                   flags);
       if (err > 0)

.

author
#6 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

In reply to Theodore Ts'o

Hi,

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

fs/ext4/extents.c +2 -0
index c0de30f25185..66767ede235f 100644
@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle,
                goto out;
 
        if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
+               int ret = 0;
+
                if (split_flag & 
(EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
                        if (split_flag & EXT4_EXT_DATA_VALID1) {
-                               err = ext4_ext_zeroout(inode, ex2);
+                               ret = ext4_ext_zeroout(inode, ex2);
                                zero_ex.ee_block = ex2->ee_block;
                                zero_ex.ee_len = cpu_to_le16(
ext4_ext_get_actual_len(ex2));
                                ext4_ext_store_pblock(&zero_ex,
ext4_ext_pblock(ex2));
                        } else {
-                               err = ext4_ext_zeroout(inode, ex);
+                               ret = ext4_ext_zeroout(inode, ex);
                                zero_ex.ee_block = ex->ee_block;
                                zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(ex));
@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle,

                                                      ext4_ext_pblock(ex));
                        }
                } else {
-                       err = ext4_ext_zeroout(inode, &orig_ex);
+                       ret = ext4_ext_zeroout(inode, &orig_ex);
                        zero_ex.ee_block = orig_ex.ee_block;
                        zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(&orig_ex));
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle,

                                              ext4_ext_pblock(&orig_ex));
                }

- if (!err) {

+               if (!ret) {
                        /* update the extent length and mark as 
initialized */
                        ex->ee_len = cpu_to_le16(ee_len);
                        ext4_ext_try_to_merge(handle, inode, path, ex);
fs/ext4/inode.c +3 -0
index d18852d6029c..95b970581864 100644
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t lblk, ext4_fsblk_t pblk,
        if (ret > 0)
                ret = 0;
 
+       if (ret == -ENOSPC)
+               ret = -EIO;
+
        return ret;
 }
 
Quoted 54 lines · Theodore Ts'o

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

---

I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?

  fs/ext4/extents.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  				split_map.m_len - ee_block);
  			err = ext4_ext_zeroout(inode, &zero_ex1);
  			if (err)
-				goto out;
+				goto fallback;
  			split_map.m_len = allocated;
  		}
  		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  						      ext4_ext_pblock(ex));
  				err = ext4_ext_zeroout(inode, &zero_ex2);
  				if (err)
-					goto out;
+					goto fallback;
  			}
 
  			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  		}
  	}
 
+fallback:
  	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
  				flags);
  	if (err > 0)
reviewer
#7 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Hello,

Quoted 10 lines · yangerkun

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?

Honza

Quoted 119 lines
fs/ext4/extents.c +4 -2
index c0de30f25185..66767ede235f 100644
@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle,
                goto out;
 
        if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
+               int ret = 0;
+
                if (split_flag &
 (EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
                        if (split_flag & EXT4_EXT_DATA_VALID1) {
-                               err = ext4_ext_zeroout(inode, ex2);
+                               ret = ext4_ext_zeroout(inode, ex2);
                                zero_ex.ee_block = ex2->ee_block;
                                zero_ex.ee_len = cpu_to_le16(
 
 ext4_ext_get_actual_len(ex2));
                                ext4_ext_store_pblock(&zero_ex,
 
 ext4_ext_pblock(ex2));
                        } else {
-                               err = ext4_ext_zeroout(inode, ex);
+                               ret = ext4_ext_zeroout(inode, ex);
                                zero_ex.ee_block = ex->ee_block;
                                zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(ex));

fs/ext4/extents.c +1 -1
@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                                      ext4_ext_pblock(ex));
                        }
                } else {
-                       err = ext4_ext_zeroout(inode, &orig_ex);
+                       ret = ext4_ext_zeroout(inode, &orig_ex);
                        zero_ex.ee_block = orig_ex.ee_block;
                        zero_ex.ee_len = cpu_to_le16(
 

ext4_ext_get_actual_len(&orig_ex));

fs/ext4/extents.c +1 -1
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                              ext4_ext_pblock(&orig_ex));
                }
 
-               if (!err) {
+               if (!ret) {
                        /* update the extent length and mark as initialized
 */
                        ex->ee_len = cpu_to_le16(ee_len);
                        ext4_ext_try_to_merge(handle, inode, path, ex);
fs/ext4/inode.c +3 -0
index d18852d6029c..95b970581864 100644
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t
 lblk, ext4_fsblk_t pblk,
        if (ret > 0)
                ret = 0;
 
+       if (ret == -ENOSPC)
+               ret = -EIO;
+
        return ret;
 }
Quoted 52 lines · Theodore Ts'o

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

---

I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?

  fs/ext4/extents.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  				split_map.m_len - ee_block);
  			err = ext4_ext_zeroout(inode, &zero_ex1);
  			if (err)
-				goto out;
+				goto fallback;
  			split_map.m_len = allocated;
  		}
  		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  						      ext4_ext_pblock(ex));
  				err = ext4_ext_zeroout(inode, &zero_ex2);
  				if (err)
-					goto out;
+					goto fallback;
  			}
  			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
  		}
  	}
+fallback:
  	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
  				flags);
  	if (err > 0)
 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author
#8 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 14 lines · Jan Kara

Hello,

Quoted 10 lines · yangerkun

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

Quoted 6 lines

I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?

I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...

Quoted 122 lines

Honza

Quoted 119 lines
fs/ext4/extents.c +4 -2
index c0de30f25185..66767ede235f 100644
@@ -3218,16 +3218,18 @@ static int ext4_split_extent_at(handle_t *handle,
                 goto out;
 
         if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
+               int ret = 0;
+
                 if (split_flag &
 (EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
                         if (split_flag & EXT4_EXT_DATA_VALID1) {
-                               err = ext4_ext_zeroout(inode, ex2);
+                               ret = ext4_ext_zeroout(inode, ex2);
                                 zero_ex.ee_block = ex2->ee_block;
                                 zero_ex.ee_len = cpu_to_le16(
 
 ext4_ext_get_actual_len(ex2));
                                 ext4_ext_store_pblock(&zero_ex,
 
 ext4_ext_pblock(ex2));
                         } else {
-                               err = ext4_ext_zeroout(inode, ex);
+                               ret = ext4_ext_zeroout(inode, ex);
                                 zero_ex.ee_block = ex->ee_block;
                                 zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(ex));

fs/ext4/extents.c +1 -1
@@ -3235,7 +3237,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                                       ext4_ext_pblock(ex));
                         }
                 } else {
-                       err = ext4_ext_zeroout(inode, &orig_ex);
+                       ret = ext4_ext_zeroout(inode, &orig_ex);
                         zero_ex.ee_block = orig_ex.ee_block;
                         zero_ex.ee_len = cpu_to_le16(
 

ext4_ext_get_actual_len(&orig_ex));

fs/ext4/extents.c +1 -1
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                               ext4_ext_pblock(&orig_ex));
                 }
 
-               if (!err) {
+               if (!ret) {
                         /* update the extent length and mark as initialized
 */
                         ex->ee_len = cpu_to_le16(ee_len);
                         ext4_ext_try_to_merge(handle, inode, path, ex);
fs/ext4/inode.c +3 -0
index d18852d6029c..95b970581864 100644
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t
 lblk, ext4_fsblk_t pblk,
         if (ret > 0)
                 ret = 0;
 
+       if (ret == -ENOSPC)
+               ret = -EIO;
+
         return ret;
  }
Quoted 52 lines · Theodore Ts'o

If the underlying storage device is using thin-provisioning, it's possible for a zeroout operation to return ENOSPC.

Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks left") added logic to retry block allocation since we might get free block after we commit a transaction. But the ENOSPC from thin-provisioning will confuse ext4, and lead to an infinite loop.

Since using zeroout instead of splitting the extent node is an optimization, if it fails, we might as well fall back to splitting the extent node.

---

I've run this through my battery of tests, and it doesn't cause any regressions. Yangerkun, can you test this and see if this works for you?

   fs/ext4/extents.c | 5 +++--
   1 file changed, 3 insertions(+), 2 deletions(-)
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
   				split_map.m_len - ee_block);
   			err = ext4_ext_zeroout(inode, &zero_ex1);
   			if (err)
-				goto out;
+				goto fallback;
   			split_map.m_len = allocated;
   		}
   		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
   						      ext4_ext_pblock(ex));
   				err = ext4_ext_zeroout(inode, &zero_ex2);
   				if (err)
-					goto out;
+					goto fallback;
   			}
   			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
   		}
   	}
+fallback:
   	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
   				flags);
   	if (err > 0)
 
reviewer
#9 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara

Hello,

Quoted 10 lines · yangerkun

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

Quoted 6 lines

I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?

Quoted 3 lines

I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...

OK, so passing back original error instead of the error from ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at() called from ext4_writepages() because we should have reserved enough space for extent splits when writing data. So hitting that is already unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug. Secondly, returning EIO instead of ENOSPC is IMO a bit confusing for upper layers and makes it harder to analyze where the real problem is...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author
#10 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 27 lines · Jan Kara
Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara

Hello,

Quoted 10 lines · yangerkun

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

Quoted 6 lines

I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?

Quoted 3 lines

I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...

Quoted 5 lines

OK, so passing back original error instead of the error from ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at() called from ext4_writepages() because we should have reserved enough space for extent splits when writing data. So hitting that is already

ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_space

It seems we will only reserve space for data, no for metadata...

Quoted 3 lines

unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug.

Agree. EIO from ext4_ext_zeroout that overwrite the ENOSPC from ext4_ext_insert_extent seems buggy too. Maybe we should ignore the error from ext4_ext_zeroout and return the error from ext4_ext_insert_extent once ext4_ext_zeroout in ext4_split_extent_at got a error. Something like this:

fs/ext4/extents.c +1 -0
index 0ecf819bf189..56cc00ee42a1 100644
@@ -3185,6 +3185,7 @@ static int ext4_split_extent_at(handle_t *handle,
        struct ext4_extent *ex2 = NULL;
        unsigned int ee_len, depth;
        int err = 0;
+       int err1;
 
        BUG_ON((split_flag & (EXT4_EXT_DATA_VALID1 | 

EXT4_EXT_DATA_VALID2)) ==

               (EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2));
@@ -3255,7 +3256,7 @@ static int ext4_split_extent_at(handle_t *handle,
        if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
                if (split_flag & 
(EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
                        if (split_flag & EXT4_EXT_DATA_VALID1) {
-                               err = ext4_ext_zeroout(inode, ex2);
+                               err1 = ext4_ext_zeroout(inode, ex2);
                                zero_ex.ee_block = ex2->ee_block;
                                zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(ex2));
@@ -3270,7 +3271,7 @@ static int ext4_split_extent_at(handle_t *handle,

                                                      ext4_ext_pblock(ex));
                        }
                } else {
-                       err = ext4_ext_zeroout(inode, &orig_ex);
+                       err1 = ext4_ext_zeroout(inode, &orig_ex);
                        zero_ex.ee_block = orig_ex.ee_block;
                        zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(&orig_ex));
@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle,

                                              ext4_ext_pblock(&orig_ex));
                }

- if (!err) {

+               if (!err1) {
                        /* update the extent length and mark as 
initialized */
                        ex->ee_len = cpu_to_le16(ee_len);
                        ext4_ext_try_to_merge(handle, inode, path, ex);
Quoted 6 lines

Secondly, returning EIO instead of ENOSPC is IMO a
bit confusing for upper layers and makes it harder to analyze where the real problem is...

Honza

reviewer
#11 lore raw

Re: [PATCH] ext4: if zeroout fails fall back to splitting the extent node

Quoted 48 lines · yangerkun
Quoted 27 lines · Jan Kara
Quoted 22 lines · yangerkun
Quoted 14 lines · Jan Kara

Hello,

Quoted 10 lines · yangerkun

Rethink about this problem. Should we consider other place which call ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not really happened)...

How about include follow patch which not only transfer ENOSPC to EIO. But also stop to overwrite the error return by ext4_ext_insert_extent in ext4_split_extent_at.

Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the extent node") can work together with this patch.

Quoted 6 lines

I've got back to this. The ext4_ext_zeroout() calls in ext4_split_extent_at() seem to be there as fallback when insertion of a new extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I think returning an error as the code does now is correct and we don't have much other option. Also we are really running out of disk space so I think returning ENOSPC is fine. What exact scenario are you afraid of?

Quoted 3 lines

I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...

Quoted 5 lines

OK, so passing back original error instead of the error from ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at() called from ext4_writepages() because we should have reserved enough space for extent splits when writing data. So hitting that is already

ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_space

It seems we will only reserve space for data, no for metadata...

Quoted 3 lines

unexpected. Committing transaction holding blocks that are expected to be free is the most likely reason for us seeing ENOSPC and returning EIO in that case would be bug.

Quoted 5 lines

Agree. EIO from ext4_ext_zeroout that overwrite the ENOSPC from ext4_ext_insert_extent seems buggy too. Maybe we should ignore the error from ext4_ext_zeroout and return the error from ext4_ext_insert_extent once ext4_ext_zeroout in ext4_split_extent_at got a error. Something like this:

Yep, something like that looks good to me.

Honza

Quoted 54 lines
fs/ext4/extents.c +1 -0
index 0ecf819bf189..56cc00ee42a1 100644
@@ -3185,6 +3185,7 @@ static int ext4_split_extent_at(handle_t *handle,
        struct ext4_extent *ex2 = NULL;
        unsigned int ee_len, depth;
        int err = 0;
+       int err1;
 
        BUG_ON((split_flag & (EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2))

==

               (EXT4_EXT_DATA_VALID1 | EXT4_EXT_DATA_VALID2));
fs/ext4/extents.c +1 -1
@@ -3255,7 +3256,7 @@ static int ext4_split_extent_at(handle_t *handle,
        if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
                if (split_flag &
 (EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
                        if (split_flag & EXT4_EXT_DATA_VALID1) {
-                               err = ext4_ext_zeroout(inode, ex2);
+                               err1 = ext4_ext_zeroout(inode, ex2);
                                zero_ex.ee_block = ex2->ee_block;
                                zero_ex.ee_len = cpu_to_le16(

ext4_ext_get_actual_len(ex2));

fs/ext4/extents.c +1 -1
@@ -3270,7 +3271,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                                      ext4_ext_pblock(ex));
                        }
                } else {
-                       err = ext4_ext_zeroout(inode, &orig_ex);
+                       err1 = ext4_ext_zeroout(inode, &orig_ex);
                        zero_ex.ee_block = orig_ex.ee_block;
                        zero_ex.ee_len = cpu_to_le16(
 

ext4_ext_get_actual_len(&orig_ex));

fs/ext4/extents.c +1 -1
@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle,
                                              ext4_ext_pblock(&orig_ex));
                }
 
-               if (!err) {
+               if (!err1) {
                        /* update the extent length and mark as initialized
 */
                        ex->ee_len = cpu_to_le16(ee_len);
                        ext4_ext_try_to_merge(handle, inode, path, ex);
Quoted 6 lines

Secondly, returning EIO instead of ENOSPC is IMO a
bit confusing for upper layers and makes it harder to analyze where the real problem is...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
reviewer
#12 lore raw
In reply to Theodore Ts'o
Quoted 22 lines · Theodore Ts'o
Quoted 21 lines · Jan Kara
Quoted 4 lines · yangerkun

Our testcase(briefly described as fsstress on dm thin-provisioning which ext4 see volume size with 100G but actual size 10G) trigger a hungtask bug since ext4_writepages fall into a infinite loop:

Quoted 9 lines

Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC

Thanks for the patch. As a quick fix for the problem this is probably fine. But longer term we might need to implement a configurable behavior for this because just dropping data on the floor (which is what would happen here) need not be what sysadmin wants and blocking until space is provisioned may be actually a preferable behavior. Anyway for now feel free to add:

Quoted 4 lines
Quoted 1 line

Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?

Yes, that looks indeed better. I'd note that even splitting extent may fail due to ENOSPC on thin-provisioned storage but the chances are *much* lower.

Honza

Quoted 30 lines
fs/ext4/extents.c +3 -2
index 92ad64b89d9b..501516cadc1b 100644
@@ -3569,7 +3569,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 				split_map.m_len - ee_block);
 			err = ext4_ext_zeroout(inode, &zero_ex1);
 			if (err)
-				goto out;
+				goto fallback;
 			split_map.m_len = allocated;
 		}
 		if (split_map.m_lblk - ee_block + split_map.m_len <
@@ -3583,7 +3583,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 						      ext4_ext_pblock(ex));
 				err = ext4_ext_zeroout(inode, &zero_ex2);
 				if (err)
-					goto out;
+					goto fallback;
 			}
 
 			split_map.m_len += split_map.m_lblk - ee_block;
@@ -3592,6 +3592,7 @@ static int ext4_ext_convert_to_initialized(handle_t *handle,
 		}
 	}
 
+fallback:
 	err = ext4_split_extent(handle, inode, ppath, &split_map, split_flag,
 				flags);
 	if (err > 0)
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#13 lore raw
Quoted 3 lines · Jan Kara

Yes, that looks indeed better. I'd note that even splitting extent may fail due to ENOSPC on thin-provisioned storage but the chances are *much* lower.

Indeed, any kind of metadata update (updating an inode atime, creating a new inode, deleting a directory entry, etc.) can fail due to ENOSPC on thin-provision storage, leading to a potentially corrupted file system since some writes will succeed, while others won't --- and that's not a scenario that's super well tested, nor is there much we can do other than remounting the file system read/only or forcing a reboot.

But at least we won't throw the kernel into an infinite loop, which is what yangerkun was reporting...

- Ted