Skip to contentSkip to conversation

ext4: fix sleep inside spinlock issue aka #14739

PATCH In review Dmitry Monakhov 4 participants 9 messages Dec 10, 2009 – Dec 19, 2009 ext4 file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ext4: fix sleep inside spinlock issue aka #14739

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

---
 fs/ext4/inode.c |   29 ++++++++++++++++-------------
 1 files changed, 16 insertions(+), 13 deletions(-)
fs/ext4/inode.c +16 -13
index 942e183..f693768 100644
@@ -1852,19 +1852,8 @@ repeat:
 	md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
 	total = md_needed + nrblocks;
 
-	/*
-	 * Make quota reservation here to prevent quota overflow
-	 * later. Real quota accounting is done at pages writeout
-	 * time.
-	 */
-	if (vfs_dq_reserve_block(inode, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-		return -EDQUOT;
-	}
-
 	if (ext4_claim_free_blocks(sbi, total)) {
 		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-		vfs_dq_release_reservation_block(inode, total);
 		if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
 			yield();
 			goto repeat;
@@ -1872,10 +1861,24 @@ repeat:
 		return -ENOSPC;
 	}
 	EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
-	EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
+	EXT4_I(inode)->i_reserved_meta_blocks += md_needed;
+	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
+
+	/*
+	 * Make quota reservation here to prevent quota overflow
+	 * later. Real quota accounting is done at pages writeout
+	 * time.
+	 */
+	if (!vfs_dq_reserve_block(inode, total))
+		return 0;       /* success */
 
+	/* Quota reservation has failed, revert inode's reservation */
+	percpu_counter_sub(&sbi->s_dirtyblocks_counter, total);
+	spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
+	EXT4_I(inode)->i_reserved_data_blocks -= nrblocks;
+	EXT4_I(inode)->i_reserved_meta_blocks -= md_needed;
 	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-	return 0;       /* success */
+	return -EDQUOT;
 }
 
 static void ext4_da_release_space(struct inode *inode, int to_free)
Signature · 1 line
1.6.0.4
reviewer
#1 lore raw
Quoted 4 lines

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

Sorry if someone already refused this (I didn't follow the previous
discussion too closely) but: Looking at the code I see no reason why ext4_claim_free_blocks needs i_block_reservation_lock. In fact mballoc calls this function without the lock. So could not we just compute 'total' under the lock, release it, reserve quota and then claim free blocks? You'd get rid of undoing the block reservation and obtain quota and blocks in the usual order...

Honza

Quoted 62 lines
---
 fs/ext4/inode.c |   29 ++++++++++++++++-------------
 1 files changed, 16 insertions(+), 13 deletions(-)
fs/ext4/inode.c +16 -13
index 942e183..f693768 100644
@@ -1852,19 +1852,8 @@ repeat:
 	md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
 	total = md_needed + nrblocks;
 
-	/*
-	 * Make quota reservation here to prevent quota overflow
-	 * later. Real quota accounting is done at pages writeout
-	 * time.
-	 */
-	if (vfs_dq_reserve_block(inode, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-		return -EDQUOT;
-	}
-
 	if (ext4_claim_free_blocks(sbi, total)) {
 		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-		vfs_dq_release_reservation_block(inode, total);
 		if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
 			yield();
 			goto repeat;
@@ -1872,10 +1861,24 @@ repeat:
 		return -ENOSPC;
 	}
 	EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
-	EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
+	EXT4_I(inode)->i_reserved_meta_blocks += md_needed;
+	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
+
+	/*
+	 * Make quota reservation here to prevent quota overflow
+	 * later. Real quota accounting is done at pages writeout
+	 * time.
+	 */
+	if (!vfs_dq_reserve_block(inode, total))
+		return 0;       /* success */
 
+	/* Quota reservation has failed, revert inode's reservation */
+	percpu_counter_sub(&sbi->s_dirtyblocks_counter, total);
+	spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
+	EXT4_I(inode)->i_reserved_data_blocks -= nrblocks;
+	EXT4_I(inode)->i_reserved_meta_blocks -= md_needed;
 	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
-	return 0;       /* success */
+	return -EDQUOT;
 }
 
 static void ext4_da_release_space(struct inode *inode, int to_free)
Signature · 6 lines
1.6.0.4

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#2 lore raw

PATCH 1/1 ext4: fix sleep inside spinlock issue aka #14739 V2

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
- simplify the patch according to Jan's comments

---
 fs/ext4/inode.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +2 -2
index 942e183..2327f7a 100644
@@ -1851,6 +1851,7 @@ repeat:
 
 	md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
 	total = md_needed + nrblocks;
+	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 
 	/*
 	 * Make quota reservation here to prevent quota overflow
@@ -1858,12 +1859,10 @@ repeat:
 	 * time.
 	 */
 	if (vfs_dq_reserve_block(inode, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		return -EDQUOT;
 	}
 
 	if (ext4_claim_free_blocks(sbi, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		vfs_dq_release_reservation_block(inode, total);
 		if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
 			yield();
@@ -1871,10 +1870,11 @@ repeat:
 		}
 		return -ENOSPC;
 	}
+	spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
 	EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
 	EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
reviewer
#3 lore raw

Re: [PATCH] ext4: fix sleep inside spinlock issue aka #14739 V2

Quoted 6 lines · Dmitry Monakhov

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
- simplify the patch according to Jan's comments

Dmitry, I suppose I should also merge this patch together with your other
fixes, right? For some reason, it was not part of the last submission of your patch series...

Honza

Quoted 46 lines
---
 fs/ext4/inode.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +3 -3
index 942e183..2327f7a 100644
@@ -1851,6 +1851,7 @@ repeat:
 
 	md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
 	total = md_needed + nrblocks;
+	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 
 	/*
 	 * Make quota reservation here to prevent quota overflow
@@ -1858,12 +1859,10 @@ repeat:
 	 * time.
 	 */
 	if (vfs_dq_reserve_block(inode, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		return -EDQUOT;
 	}
 
 	if (ext4_claim_free_blocks(sbi, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		vfs_dq_release_reservation_block(inode, total);
 		if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
 			yield();
@@ -1871,10 +1870,11 @@ repeat:
 		}
 		return -ENOSPC;
 	}
+	spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
 	EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
 	EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
-
 	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
+
 	return 0;       /* success */
 }
 
Signature · 1 line
1.6.0.4
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw

Re: [PATCH] ext4: fix sleep inside spinlock issue aka #14739 V2

2009/12/16 Jan Kara <[email protected]>:

Quoted 10 lines · Dmitry Monakhov
Quoted 6 lines

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
 - simplify the patch according to Jan's comments

 Dmitry, I suppose I should also merge this patch together with your other fixes, right? For some reason, it was not part of the last submission of your patch series...

yes. it is not in the series because it is another bug, and it is not depended from
other patches.
Please merge newst version of the patch http://patchwork.ozlabs.org/patch/40896/
This version is affected by stupid bug, see at the bottom of the patch. In fact this patch does help Justin Maggard to overcome his issue.

BTW i've also have another patch http://patchwork.ozlabs.org/patch/40805/
it was acked by you, but ASAIK it was't pushed to the ext4 queue yet.

Quoted 28 lines

                                                               Honza

Quoted 25 lines

---
 fs/ext4/inode.c |    6 +++---

 1 files changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +1 -1
index 942e183..2327f7a 100644
@@ -1851,6 +1851,7 @@ repeat:
 
       md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
       total = md_needed + nrblocks;
+     spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 
       /*
        * Make quota reservation here to prevent quota overflow
@@ -1858,12 +1859,10 @@ repeat:
        * time.
        */
       if (vfs_dq_reserve_block(inode, total)) {
-             spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
               return -EDQUOT;
       }
 
Quoted 12 lines

fs/ext4/inode.c repeat: · line 1875

Quoted 12 lines
fs/ext4/inode.c +1 -1
       if (ext4_claim_free_blocks(sbi, total)) {
-             spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
               vfs_dq_release_reservation_block(inode, total);
               if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
                       yield();
@@ -1871,10 +1870,11 @@ repeat:
               }
               return -ENOSPC;
       }
+     spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
       EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
       EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta_blocks may be changed after we dropped the lock, so we have to use add here:
EXT4_I(inode)->i_reserved_meta_blocks += mb_needed;
I've overlooked this simple bug, and in fact i'm able to catch it only with havy io load
./fsstress -p16 .....

Quoted 6 lines
Quoted 6 lines
fs/ext4/inode.c +1 -1
-
       spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
+

      return 0;       /* success */
 }

Quoted 7 lines
Quoted 3 lines

--
1.6.0.4

--
Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

reviewer
#5 lore raw

Re: [PATCH] ext4: fix sleep inside spinlock issue aka #14739 V2

Quoted 18 lines · Dmitry Monakhov

2009/12/16 Jan Kara <[email protected]>:

Quoted 10 lines · Dmitry Monakhov
Quoted 6 lines

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
 - simplify the patch according to Jan's comments

 Dmitry, I suppose I should also merge this patch together with your other fixes, right? For some reason, it was not part of the last submission of your patch series...

yes. it is not in the series because it is another bug, and it is not depended from
other patches.
Please merge newst version of the patch http://patchwork.ozlabs.org/patch/40896/
This version is affected by stupid bug, see at the bottom of the patch. In fact this patch does help Justin Maggard to overcome his issue.

Quoted 2 lines

BTW i've also have another patch http://patchwork.ozlabs.org/patch/40805/
it was acked by you, but ASAIK it was't pushed to the ext4 queue yet.

OK, I've added both patches to my tree. Usually Ted takes care of ext4
fixes but given that I carry also other ext4 quota fixes, I think I can merge these as well if he does not object.

Honza

Signature · 6 lines
Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#6 lore raw

Re: [PATCH] ext4: fix sleep inside spinlock issue aka #14739 V2

In reply to Dmitry Monakhov
Quoted 38 lines · Dmitry Monakhov

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
- simplify the patch according to Jan's comments

---
 fs/ext4/inode.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +1 -2
index 942e183..2327f7a 100644
@@ -1851,6 +1851,7 @@ repeat:
 
 	md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
 	total = md_needed + nrblocks;
+	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 
 	/*
 	 * Make quota reservation here to prevent quota overflow
@@ -1858,12 +1859,10 @@ repeat:
 	 * time.
 	 */
 	if (vfs_dq_reserve_block(inode, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		return -EDQUOT;
 	}
 
 	if (ext4_claim_free_blocks(sbi, total)) {
-		spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 		vfs_dq_release_reservation_block(inode, total);
 		if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
 			yield();

fs/ext4/inode.c repeat: · line 1878

Quoted 10 lines
fs/ext4/inode.c +2 -1
@@ -1871,10 +1870,11 @@ repeat:
 		}
 		return -ENOSPC;
 	}
+	spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
 	EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
 	EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
-
 	spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
+
 	return 0;       /* success */

NACK
I guess we would end up setting i_reserved_meta_blocks wrongly because mdblocks could be based on the old value of i_reserved_meta_blocks because we are dropping i_block_reservation_lock lock.

-aneesh

#7 lore raw

Re: [PATCH] ext4: fix sleep inside spinlock issue aka #14739 V2

2009/12/18 Aneesh Kumar K.V <[email protected]>:

Quoted 47 lines · Dmitry Monakhov
Quoted 46 lines

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

changes from previous version:
 - simplify the patch according to Jan's comments

---
 fs/ext4/inode.c |    6 +++---

 1 files changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +2 -3
index 942e183..2327f7a 100644
@@ -1851,6 +1851,7 @@ repeat:
 
       md_needed = mdblocks - EXT4_I(inode)->i_reserved_meta_blocks;
       total = md_needed + nrblocks;
+     spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
 
       /*
        * Make quota reservation here to prevent quota overflow
@@ -1858,12 +1859,10 @@ repeat:
        * time.
        */
       if (vfs_dq_reserve_block(inode, total)) {
-             spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
               return -EDQUOT;
       }
 
       if (ext4_claim_free_blocks(sbi, total)) {
-             spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
               vfs_dq_release_reservation_block(inode, total);
               if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
                       yield();
@@ -1871,10 +1870,11 @@ repeat:
               }
               return -ENOSPC;
       }
+     spin_lock(&EXT4_I(inode)->i_block_reservation_lock);
       EXT4_I(inode)->i_reserved_data_blocks += nrblocks;
       EXT4_I(inode)->i_reserved_meta_blocks = mdblocks;
-
       spin_unlock(&EXT4_I(inode)->i_block_reservation_lock);
Quoted 8 lines
Quoted 2 lines
fs/ext4/inode.c +1 -0
+

      return 0;       /* success */

NACK
I guess we would end up setting i_reserved_meta_blocks wrongly because mdblocks could be based on the old value of i_reserved_meta_blocks because we are dropping i_block_reservation_lock lock.

Yes. you right. I've already fixed this.
Please take a look at committed version:
http://git.kernel.org/?p=linux/kernel/git/jack/linux-fs-2.6.git;a=commitdiff;h=ef22d6deda461ef32c72944f662863c022253571

Quoted 3 lines

-aneesh

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
In reply to Jan Kara
Quoted 11 lines · Jan Kara
Quoted 4 lines

drop i_block_reservation_lock before vfs_dq_reserve_block(). this patch fix http://bugzilla.kernel.org/show_bug.cgi?id=14739

Sorry if someone already refused this (I didn't follow the previous
discussion too closely) but: Looking at the code I see no reason why ext4_claim_free_blocks needs i_block_reservation_lock. In fact mballoc calls this function without the lock. So could not we just compute 'total' under the lock, release it, reserve quota and then claim free blocks? You'd get rid of undoing the block reservation and obtain quota and blocks in the usual order...

The code is protecting i_reserved_meta_blocks. We are recalculating the the value and need to make sure we don't get the value wrong. I guess we need to hold i_block_reservation_lock while we recalculate the meta data block.

-aneesh