Lack of mutex_lock in drop_pagecache_sb()
Subsystems (3)
PATCH 1/1 Lack of mutex_lock in drop_pagecache_sb()
I create the patch which fixes lack of mutex_lock in drop_pagecache_sb(). Please check the bug and the patch (below).
----------------------------------------------------------------------
When drop_pagecache_sb() frees inodes, it doesn't get mutex_lock of iprune_mutex. Therefore, if it races the process which frees inodes (ex. prune_icache()), OS panic may happen.
An example of the panic flow is the following:
----------------------------------------------------------------------
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | | | V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | | | | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |
(time)
----------------------------------------------------------------------
If the inode which Process B do list_move() with is the same as the one which Process A did list_del() with, OS may panic.
Fixing this bug requires the following:
- Adding mutex_lock of iprune_mutex to drop_pagecache_sb().
- Changing iprune_mutex from static to global.
--- fs/drop_caches.c | 2 ++ fs/inode.c | 2 +- include/linux/writeback.h | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
fs/drop_caches.c 2009-03-11 04:04:29.705284864 +0900 renamed from fs/drop_caches.c 2009-03-10 06:38:17.137283010 +0900 +2 -0
@@ -16,6 +16,7 @@ static void drop_pagecache_sb(struct sup { struct inode *inode, *toput_inode = NULL; + mutex_lock(&iprune_mutex); spin_lock(&inode_lock); list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { if (inode->i_state & (I_FREEING|I_WILL_FREE))
@@ -31,6 +32,7 @@ static void drop_pagecache_sb(struct sup } spin_unlock(&inode_lock); iput(toput_inode); + mutex_unlock(&iprune_mutex); } static void drop_pagecache(void)
fs/inode.c 2009-03-18 07:17:31.032284423 +0900 renamed from fs/inode.c 2009-03-10 06:38:22.404282773 +0900 +1 -1
@@ -91,7 +91,7 @@ DEFINE_SPINLOCK(inode_lock); * from its final dispose_list, the struct super_block they refer to * (for inode->i_sb->s_op) may already have been freed and reused. */ -static DEFINE_MUTEX(iprune_mutex); +DEFINE_MUTEX(iprune_mutex); /* * Statistics gathering..
include/linux/writeback.h 2009-03-11 05:27:43.679283477 +0900 renamed from include/linux/writeback.h 2009-03-10 06:38:27.133285255 +0900 +1 -0
@@ -10,6 +10,7 @@ struct backing_dev_info; extern spinlock_t inode_lock; +extern struct mutex iprune_mutex; extern struct list_head inode_in_use; extern struct list_head inode_unused;
Masasyoshi,
Quoted 2 lines · Masasyoshi MIZUMA
I create the patch which fixes lack of
mutex_lockindrop_pagecache_sb(). Please check the bug and the patch (below).
Is this a real producible bug or a theory one?
IMHO the I_FREEING flag should avoid the race.
Quoted 8 lines
----------------------------------------------------------------------
When
drop_pagecache_sb()frees inodes, it doesn't getmutex_lockofiprune_mutex. Therefore, if it races the process which frees inodes (ex.prune_icache()), OS panic may happen.An example of the panic flow is the following:
----------------------------------------------------------------------
Quoted 11 lines
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | |
inode->i_state |= I_FREEING;
Quoted 3 lines
| V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | |
if (inode->i_state & (I_FREEING|I_WILL_FREE))
continue;Quoted 9 lines
| | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |
Quoted 4 lines
(time)
----------------------------------------------------------------------
If the inode which Process B dolist_move()with is the same as the one which Process A didlist_del()with, OS may panic.
Thanks,
Fengguang
Hi, Fengguang
Quoted 6 lines · On Mon, 23 Mar 2009 18:38:46 +0800 Wu Fengguang <[email protected]> wrote:
Masasyoshi,
Quoted 2 lines · Masasyoshi MIZUMA
I create the patch which fixes lack of
mutex_lockindrop_pagecache_sb(). Please check the bug and the patch (below).
Thank you for your comment, and I apologize to you for my lack of explanation.
Quoted 1 line
Is this a real producible bug or a theory one?
This is a real bug.
Quoted 1 line
IMHO the
I_FREEINGflag should avoid the race.
I supplement the explanation for this problem.
clear_inode() is called by dispose_list(), and sets the inode's i_state to I_CLEAR. Therefore, the following conditional expression doesn't match for the inode:
"if (inode->i_state & (I_FREEING|I_WILL_FREE)) continue;" As the result, this problem can happen.
Quoted 21 lines
Quoted 9 lines
----------------------------------------------------------------------
When
drop_pagecache_sb()frees inodes, it doesn't getmutex_lockofiprune_mutex. Therefore, if it races the process which frees inodes (ex.prune_icache()), OS panic may happen.An example of the panic flow is the following:
----------------------------------------------------------------------Quoted 11 lines
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | |
Quoted 22 lines
inode->i_state |= I_FREEING;Quoted 3 lines
| V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | |if (inode->i_state & (I_FREEING|I_WILL_FREE)) continue;Quoted 13 lines
| | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |(time)
----------------------------------------------------------------------
If the inode which Process B dolist_move()with is the same as the one which Process A didlist_del()with, OS may panic.
I applied your comment and then modified the panic flow figure. Please check below:
----------------------------------------------------------------------
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | | | V | | | inode->i_state |= I_FREEING; | | | | | | | V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | | | | | | | V | | | dispose_list() | | | list_del() | | | | | | | V | | | clear_inode() | | | inode->i_state = I_CLEAR | | | | | | | | | V | | | if (inode->i_state & (I_FREEING|I_WILL_FREE)) | | | continue; <---- NOT MATCH | | | | | | | V | | | __iget() | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |
(time)
----------------------------------------------------------------------
Thanks,
Masayoshi MIZUMA
Hi Masayoshi,
Quoted 110 lines · Masayoshi MIZUMA
Hi, Fengguang
Quoted 6 lines · On Mon, 23 Mar 2009 18:38:46 +0800 Wu Fengguang <[email protected]> wrote:
Masasyoshi,
Quoted 2 lines · Masasyoshi MIZUMA
I create the patch which fixes lack of
mutex_lockindrop_pagecache_sb(). Please check the bug and the patch (below).Thank you for your comment, and I apologize to you for my lack of explanation.
Quoted 1 line
Is this a real producible bug or a theory one?
This is a real bug.
Quoted 1 line
IMHO the
I_FREEINGflag should avoid the race.I supplement the explanation for this problem.
clear_inode()is called bydispose_list(), and sets the inode'si_statetoI_CLEAR. Therefore, the following conditional expression doesn't match for the inode:
"if (inode->i_state& (I_FREEING|I_WILL_FREE)) continue;" As the result, this problem can happen.Quoted 21 lines
Quoted 9 lines
----------------------------------------------------------------------
When
drop_pagecache_sb()frees inodes, it doesn't getmutex_lockofiprune_mutex. Therefore, if it races the process which frees inodes (ex.prune_icache()), OS panic may happen.An example of the panic flow is the following:
----------------------------------------------------------------------Quoted 11 lines
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | |Quoted 22 lines
inode->i_state |= I_FREEING;Quoted 3 lines
| V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | |if (inode->i_state & (I_FREEING|I_WILL_FREE)) continue;Quoted 13 lines
| | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |(time)
----------------------------------------------------------------------
If the inode which Process B dolist_move()with is the same as the one which Process A didlist_del()with, OS may panic.I applied your comment and then modified the panic flow figure. Please check below:
----------------------------------------------------------------------[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | | | V | | | inode->i_state |= I_FREEING; | | | | | | | V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | | | | | | | V | | | dispose_list() | | | list_del() | | | | | | | V | | | clear_inode() | | | inode->i_state = I_CLEAR | | | | | | | | | V | | | if (inode->i_state & (I_FREEING|I_WILL_FREE)) | | | continue; <---- NOT MATCH | | | | | | | V | | | __iget() | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |
Quoted 2 lines
(time)
----------------------------------------------------------------------
Ah thanks for the explanation!
How about this lightweight fix? Since s_umount is already taken in drop_pagecache(), it's not necessary to take iprune_mutex again.
Thanks,
Fengguang
--- fs/drop_caches.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/drop_caches.c +1 -1
@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;
Jan Kara
maintainerQuoted 116 lines · Wu Fengguang
Hi Masayoshi,
Quoted 110 lines · Masayoshi MIZUMA
Hi, Fengguang
Quoted 6 lines · On Mon, 23 Mar 2009 18:38:46 +0800 Wu Fengguang <[email protected]> wrote:
Masasyoshi,
Quoted 2 lines · Masasyoshi MIZUMA
I create the patch which fixes lack of
mutex_lockindrop_pagecache_sb(). Please check the bug and the patch (below).Thank you for your comment, and I apologize to you for my lack of explanation.
Quoted 1 line
Is this a real producible bug or a theory one?
This is a real bug.
Quoted 1 line
IMHO the
I_FREEINGflag should avoid the race.I supplement the explanation for this problem.
clear_inode()is called bydispose_list(), and sets the inode'si_statetoI_CLEAR. Therefore, the following conditional expression doesn't match for the inode:
"if (inode->i_state& (I_FREEING|I_WILL_FREE)) continue;" As the result, this problem can happen.Quoted 21 lines
Quoted 9 lines
----------------------------------------------------------------------
When
drop_pagecache_sb()frees inodes, it doesn't getmutex_lockofiprune_mutex. Therefore, if it races the process which frees inodes (ex.prune_icache()), OS panic may happen.An example of the panic flow is the following:
----------------------------------------------------------------------Quoted 11 lines
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | |Quoted 22 lines
inode->i_state |= I_FREEING;Quoted 3 lines
| V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | |if (inode->i_state & (I_FREEING|I_WILL_FREE)) continue;Quoted 13 lines
| | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |(time)
----------------------------------------------------------------------
If the inode which Process B dolist_move()with is the same as the one which Process A didlist_del()with, OS may panic.I applied your comment and then modified the panic flow figure. Please check below:
----------------------------------------------------------------------[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | | | V | | | inode->i_state |= I_FREEING; | | | | | | | V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | | | | | | | V | | | dispose_list() | | | list_del() | | | | | | | V | | | clear_inode() | | | inode->i_state = I_CLEAR | | | | | | | | | V | | | if (inode->i_state & (I_FREEING|I_WILL_FREE)) | | | continue; <---- NOT MATCH | | | | | | | V | | | __iget() | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |Quoted 2 lines
(time)
----------------------------------------------------------------------
Quoted 4 lines
Ah thanks for the explanation!
How about this lightweight fix? Since
s_umountis already taken indrop_pagecache(), it's not necessary to takeiprune_mutexagain.
Yes, the patch looks fine like this. But I believe there are more places
which might miss similar check. Probably dquot.c: add_dquot_ref() needs the same check and fs-writeback.c: generic_sync_sb_inodes() also needs adding I_CLEAR to the test, doesn't it?
Honza
Quoted 11 lines
fs/drop_caches.c +1
-1@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 117 lines · Jan Kara
Quoted 116 lines · Wu Fengguang
Hi Masayoshi,
Quoted 112 lines · Masayoshi MIZUMA
Hi, Fengguang
Quoted 6 lines · On Mon, 23 Mar 2009 18:38:46 +0800 Wu Fengguang <[email protected]> wrote:
Masasyoshi,
Quoted 2 lines · Masasyoshi MIZUMA
I create the patch which fixes lack of
mutex_lockindrop_pagecache_sb(). Please check the bug and the patch (below).Thank you for your comment, and I apologize to you for my lack of explanation.
Quoted 1 line
Is this a real producible bug or a theory one?
This is a real bug.
Quoted 1 line
IMHO the
I_FREEINGflag should avoid the race.I supplement the explanation for this problem.
clear_inode()is called bydispose_list(), and sets the inode'si_statetoI_CLEAR. Therefore, the following conditional expression doesn't match for the inode:
"if (inode->i_state& (I_FREEING|I_WILL_FREE)) continue;" As the result, this problem can happen.Quoted 21 lines
Quoted 9 lines
----------------------------------------------------------------------
When
drop_pagecache_sb()frees inodes, it doesn't getmutex_lockofiprune_mutex. Therefore, if it races the process which frees inodes (ex.prune_icache()), OS panic may happen.An example of the panic flow is the following:
----------------------------------------------------------------------Quoted 11 lines
[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | |Quoted 22 lines
inode->i_state |= I_FREEING;Quoted 3 lines
| V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | |if (inode->i_state & (I_FREEING|I_WILL_FREE)) continue;Quoted 13 lines
| | | | | V | V | dispose_list() | __iget() | list_del() | | | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |(time)
----------------------------------------------------------------------
If the inode which Process B dolist_move()with is the same as the one which Process A didlist_del()with, OS may panic.I applied your comment and then modified the panic flow figure. Please check below:
----------------------------------------------------------------------[process A] | [process B] | | | shrink_icache_memory() | | | | | V | | prune_icache() | drop_pagecache() | mutex_lock(&iprune_mutex) | | | spin_lock(&inode_lock) | | | | | V | | | drop_pagecache_sb() | | | | | V | | | inode->i_state |= I_FREEING; | | | | | | | V | V | spin_unlock(&inode_lock) | spin_lock(&inode_lock) | | | | | | | | | V | | | dispose_list() | | | list_del() | | | | | | | V | | | clear_inode() | | | inode->i_state = I_CLEAR | | | | | | | | | V | | | if (inode->i_state & (I_FREEING|I_WILL_FREE)) | | | continue; <---- NOT MATCH | | | | | | | V | | | __iget() | | | | | V | V | spin_lock(&inode_lock) | list_move() <----- PANIC !! | | V |(time)
----------------------------------------------------------------------
Quoted 8 lines
Quoted 4 lines
Ah thanks for the explanation!
How about this lightweight fix? Since
s_umountis already taken indrop_pagecache(), it's not necessary to takeiprune_mutexagain.Yes, the patch looks fine like this. But I believe there are more places
which might miss similar check. Probablydquot.c:add_dquot_ref()needs the same check andfs-writeback.c:generic_sync_sb_inodes()also needs addingI_CLEARto the test, doesn't it?
Exactly! Basically any test on I_FREEING shall be coupled with I_CLEAR, because of the I_FREEING => I_CLEAR transition _outside_ of inode_lock.
Thanks,
Fengguang
Quoted 14 lines
Quoted 11 lines
fs/drop_caches.c +1
-1@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 1/1 skip I_CLEAR state inodes
In reply to Jan KaraAdd I_CLEAR tests to drop_pagecache_sb(), generic_sync_sb_inodes() and add_dquot_ref().
clear_inode() will switch inode state from I_FREEING to I_CLEAR, and do so _outside_ of inode_lock. So any I_FREEING testing is incomplete without the testing of I_CLEAR.
Masayoshi MIZUMA first discovered the bug in drop_pagecache_sb() and Jan Kara reminds fixing the other two cases. Thanks!
--- fs/dquot.c | 2 +- fs/drop_caches.c | 2 +- fs/fs-writeback.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-)
fs/drop_caches.c +1 -1
@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;
fs/fs-writeback.c +2 -1
@@ -538,7 +538,8 @@ void generic_sync_sb_inodes(struct superlist_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & + (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )
fs/dquot.c +1 -1
@@ -793,7 +793,7 @@ static void add_dquot_ref(struct super_bcontinue ;if (!dqinit_needed (inode, type))continue ; -if (inode->i_state & (I_FREEING |I_WILL_FREE )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE ))continue ;__iget (inode);
PATCH RESEND 1/1 skip I_CLEAR state inodes
clear_inode() will switch inode state from I_FREEING to I_CLEAR, and do so _outside_ of inode_lock. So any I_FREEING testing is incomplete without a coupled testing of I_CLEAR.
So add I_CLEAR tests to drop_pagecache_sb(), generic_sync_sb_inodes() and add_dquot_ref().
Masayoshi MIZUMA discovered the bug in drop_pagecache_sb() and Jan Kara reminds fixing the other two cases.
--- fs/drop_caches.c | 2 +- fs/fs-writeback.c | 3 ++- fs/quota/dquot.c | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-)
fs/drop_caches.c +1 -1
@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;
fs/fs-writeback.c +2 -1
@@ -538,7 +538,8 @@ void generic_sync_sb_inodes(struct superlist_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & + (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )
fs/quota/dquot.c +1 -1
@@ -823,7 +823,7 @@ static void add_dquot_ref(struct super_bspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (!atomic_read (&inode->i_writecount))continue ;
Re: [PATCH][RESEND for 2.6.29-rc8-mm1] skip I_CLEAR state inodes
Quoted 9 lines · On Mon, 30 Mar 2009 15:18:24 +0800 Wu Fengguang <[email protected]> wrote:
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without a coupled testing ofI_CLEAR.So add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Masayoshi MIZUMA discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases.
ok...
But what is the user-visible consequence of this? You cc'ed [email protected] so I assume it's serious. People will want to know what problem we're fixing!
Quoted 35 lines
fs/drop_caches.c +1
-1@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;fs/fs-writeback.c +2
-1@@ -538,7 +538,8 @@ void generic_sync_sb_inodes(struct superlist_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & + (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )fs/quota/dquot.c +1
-1@@ -823,7 +823,7 @@ static void add_dquot_ref(struct super_bspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (!atomic_read (&inode->i_writecount))continue ;
Re: [PATCH][RESEND for 2.6.29-rc8-mm1] skip I_CLEAR state inodes
Quoted 10 lines · Andrew Morton
Quoted 7 lines · On Mon, 30 Mar 2009 15:18:24 +0800 Wu Fengguang <[email protected]> wrote:
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without a coupled testing ofI_CLEAR.So add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().
Quoted 8 lines
Quoted 2 lines
Masayoshi MIZUMA discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases.ok...
But what is the user-visible consequence of this? You cc'ed [email protected] so I assume it's serious. People will want to know what problem we're fixing!
Sorry, the changelog could be expanded with the following paragraph:
Fix real kernel panics. Masayoshi MIZUMA has a nice panic flow:
----------------------------------------------------------------------
[process A] | [process B] | | | prune_icache() | drop_pagecache() | spin_lock(&inode_lock) | drop_pagecache_sb() | inode->i_state |= I_FREEING; | | | spin_unlock(&inode_lock) | V | | | spin_lock(&inode_lock) | V | | | dispose_list() | | | list_del() | | | clear_inode() | | | inode->i_state = I_CLEAR | | | | | V | | | if (inode->i_state & (I_FREEING|I_WILL_FREE)) | | | continue; <==== NOT MATCH | | | | | | (DANGER from here on! Accessing disposing inode!) | | | | | | __iget() | | | list_move() <===== PANIC on poisoned list !! V V |
(time)
----------------------------------------------------------------------
Thanks,
Fengguang
Quoted 35 lines
Quoted 35 lines
fs/drop_caches.c +1
-1@@ -18,7 +18,7 @@ static void drop_pagecache_sb(struct supspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;fs/fs-writeback.c +2
-1@@ -538,7 +538,8 @@ void generic_sync_sb_inodes(struct superlist_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & + (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )fs/quota/dquot.c +1
-1@@ -823,7 +823,7 @@ static void add_dquot_ref(struct super_bspin_lock (&inode_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE |I_NEW ))continue ;if (!atomic_read (&inode->i_writecount))continue ;
Re: [PATCH] skip I_CLEAR state inodes
In reply to Wu FengguangQuoted 9 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!
Is there a reason it's not done for __sync_single_inode as well?
Jeff Layton asked the question and I'm following it up :)
__sync_single_inode currently only tests I_FREEING, but I think we are safe because __sync_single_inode sets I_SYNC, and clear_inode waits for I_SYNC to be cleared before it changes I_STATE.
On the other hand, testing I_CLEAR here probably would be safe anyway, and it'd be bonus points for consistency?
Same basic question for generic_sync_sb_inodes, which has a BUG_ON(inode->i_state & I_FREEING), seems like this could check I_CLWAR as well?
Thanks,
-Eric
Re: [PATCH] skip I_CLEAR state inodes
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().
Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?
It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.
But I_SYNC is removed just before the I_FREEING test, so we still have a small race window?
Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?
So let's add the I_CLEAR test?
Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?
Yes, we can add I_CLEAR here to catch more error condition.
Thanks,
Fengguang
---
skip I_CLEAR state inodes in writeback routines
The I_FREEING test in __sync_single_inode() is racy because clear_inode() can set i_state to I_CLEAR between the clear of I_SYNC and the test of I_FREEING.
Also extend the coverage of BUG_ON(I_FREEING) to I_CLEAR.
--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/fs-writeback.c +2 -2
@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/*
@@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ; -BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);
Jeff Layton
Acked-byRe: [PATCH] skip I_CLEAR state inodes
Quoted 16 lines · On Tue, 2 Jun 2009 16:55:23 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 9 lines
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?
Yes, I think so. __sync_single_inode clears I_SYNC but doesn't wake up the wait queue until the end of the function. So I think it's possible (though unlikely) that another thread can race in and change the state to I_CLEAR before the I_FREEING check.
Quoted 42 lines
Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/*
fs/fs-writeback.c void generic_sync_sb_inodes(struct super · line 524
Quoted 8 lines
fs/fs-writeback.c +1
-1@@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ; -BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);
Jan Kara
maintainerRe: [PATCH] skip I_CLEAR state inodes
In reply to Wu FengguangQuoted 58 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 7 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {
Is the whole if needed? I had an impression that everyone calling__sync_single_inode() should better take care it does not race with inode freeing... So WARN_ON would be more appropriate IMHO.
fs/fs-writeback.c void generic_sync_sb_inodes(struct super · line 524
Quoted 10 lines
fs/fs-writeback.c +1
-1/* @@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ; -BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);
Looking at this code, it looks a bit suspicious. What prevents this s_io
list scan to race with inode freeing? In particular generic_forget_inode() can drop inode_lock to write the inode and in the mean time generic_sync_sb_inodes() can come, get a reference to the inode and start it's writeback... Subsequent iput() would then call generic_forget_inode() on the inode again. So shouldn't we skip I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW inodes in this scan like we do for later in the function for another scan?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH] skip I_CLEAR state inodes
Quoted 15 lines · Jan Kara
Quoted 14 lines · Wu Fengguang
Quoted 11 lines · Eric Sandeen
Quoted 9 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?
yep that's right.
inode->i_state &= ~I_SYNC;
>>> clear_inode->inode_sync_wait here and find it clear <<<
if (!(inode->i_state & I_FREEING)) {...
Quoted 6 lines
Quoted 6 lines
fs/fs-writeback.c +0
-0@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ;
Quoted 8 lines
fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 4 lines
fs/fs-writeback.c +1
-1-if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.
Maybe both then (both a WARN on and then the test (defensive here, I guess)) because if we continue we may wander into a poisoned list pointer and explode, right?
-Eric
Re: [PATCH] skip I_CLEAR state inodes
Quoted 47 lines · On Tue, 02 Jun 2009 16:48:57 -0500 Eric Sandeen <[email protected]> wrote:
Quoted 15 lines · Jan Kara
Quoted 14 lines · Wu Fengguang
Quoted 11 lines · Eric Sandeen
Quoted 9 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?yep that's right.
inode->i_state &= ~I_SYNC; >>> clear_inode->inode_sync_wait here and find it clear <<< if (!(inode->i_state & I_FREEING)) {...
Quoted 6 lines
Quoted 6 lines
fs/fs-writeback.c +0
-0@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ;Quoted 8 lines
fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 4 lines
fs/fs-writeback.c +1
-1-if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.
Quoted 4 lines
Maybe both then (both a WARN on and then the test (defensive here, I guess)) because if we continue we may wander into a poisoned list pointer and explode, right?
Right.
I think removing this check would be roughly equivalent to adding a BUG_ON. It just wouldn't reliably pop since it would depend on the timing of the race.
Keeping the check and adding a WARN seems like a reasonable thing to do. Maybe after a few releases if no one has seen the WARN fire we can look at removing the check (or maybe turn it into a BUG)?
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH] skip I_CLEAR state inodes
In reply to Jan Kara[reply the easy part first]
Quoted 56 lines · Jan Kara
Quoted 55 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 13 lines
fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 10 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.
The caller doesn't matter here, because we temporarily dropped inode_lock in __sync_single_inode() ?
Jan Kara
maintainerRe: [PATCH] skip I_CLEAR state inodes
Quoted 72 lines · Wu Fengguang
[reply the easy part first]
Quoted 56 lines · Jan Kara
Quoted 55 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 13 lines
fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 10 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.
Quoted 2 lines
The caller doesn't matter here, because we temporarily dropped
inode_lockin__sync_single_inode()?
But the problem is basically what I described in the part you cut from
the email - if the caller of __sync_single_inode() does not have a reference to the inode or some other means of protecting inode from being released, then we have a problem because e.g. __writeback_single_inode() can sleep and before I_SYNC is set, all the clear_inode() can finish and we end up calling do_writepages() on invalidated mapping.
So in my opinion everybody calling __writeback_single_inode() and thus__sync_single_inode() should do an equivalent of igrab() or be sure inode cannot be freed e.g. because we have a file open.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH] skip I_CLEAR state inodes
In reply to Jan KaraQuoted 75 lines · Jan Kara
Quoted 58 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 7 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.Quoted 5 lines
fs/fs-writeback.c +0
-0/* @@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ;
Quoted 7 lines
fs/fs-writeback.c void generic_sync_sb_inodes(struct super · line 524
Quoted 5 lines
fs/fs-writeback.c +1
-1-BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);Looking at this code, it looks a bit suspicious. What prevents this
s_io
list scan to race with inode freeing? In particulargeneric_forget_inode()
Good catch.
Quoted 3 lines
can drop
inode_lockto write the inode and in the mean timegeneric_sync_sb_inodes()can come, get a reference to the inode and start it's writeback... Subsequentiput()would then callgeneric_forget_inode()
Another possibility:
generic_forget_inode
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode_lock);
generic_sync_sb_inodes()
spin_lock(&inode_lock);
__iget(inode);
__writeback_single_inode
// see non zero i_count
WARN_ON(inode->i_state & I_WILL_FREE);I'm wondering why didn't we saw reports on the last WARN_ON()? Did we missed something?
Quoted 2 lines
on the inode again. So shouldn't we skip I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW inodes in this scan like we do for later in the function for another scan?
Jan Kara
maintainerRe: [PATCH] skip I_CLEAR state inodes
Quoted 102 lines · Wu Fengguang
Quoted 75 lines · Jan Kara
Quoted 58 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 7 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.Quoted 5 lines
fs/fs-writeback.c +0
-0/* @@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ;Quoted 7 lines
fs/fs-writeback.c void generic_sync_sb_inodes(struct super · line 524
Quoted 5 lines
fs/fs-writeback.c +1
-1-BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);Looking at this code, it looks a bit suspicious. What prevents this
s_io
list scan to race with inode freeing? In particulargeneric_forget_inode()Good catch.
Quoted 3 lines
can drop
inode_lockto write the inode and in the mean timegeneric_sync_sb_inodes()can come, get a reference to the inode and start it's writeback... Subsequentiput()would then callgeneric_forget_inode()Another possibility:
generic_forget_inode
inode->i_state|=I_WILL_FREE;spin_unlock(&inode_lock); generic_sync_sb_inodes() spin_lock(&inode_lock); __iget(inode); __writeback_single_inode // see non zero i_count WARN_ON(inode->i_state & I_WILL_FREE);
Quoted 2 lines
I'm wondering why didn't we saw reports on the last
WARN_ON()? Did we missed something?
I meant the above race in my description ;-). Anyway, the race can happen
only if we are unmounting the filesystem (normally, we bail out on sb->s_flags & MS_ACTIVE check - yes, it's a bit hidden and it also took me a while to understand why we weren't seeing tons of warnings...).
Quoted 2 lines
Quoted 2 lines
on the inode again. So shouldn't we skip I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW inodes in this scan like we do for later in the function for another scan?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH] skip I_CLEAR state inodes
Quoted 103 lines · Jan Kara
Quoted 102 lines · Wu Fengguang
Quoted 75 lines · Jan Kara
Quoted 58 lines · Wu Fengguang
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · Wu Fengguang
Add
I_CLEARtests todrop_pagecache_sb(),generic_sync_sb_inodes()andadd_dquot_ref().Quoted 8 lines
Quoted 6 lines
clear_inode()will switch inode state fromI_FREEINGtoI_CLEAR, and do so _outside_ ofinode_lock. So anyI_FREEINGtesting is incomplete without the testing ofI_CLEAR.Masayoshi MIZUMA first discovered the bug in
drop_pagecache_sb()and Jan Kara reminds fixing the other two cases. Thanks!Is there a reason it's not done for
__sync_single_inodeas well?It missed the glance because it don't have an obvious '|' in the line ;)
Quoted 5 lines
Jeff Layton asked the question and I'm following it up :)
__sync_single_inodecurrently only testsI_FREEING, but I think we are safe because__sync_single_inodesetsI_SYNC, andclear_inodewaits forI_SYNCto be cleared before it changesI_STATE.But
I_SYNCis removed just before theI_FREEINGtest, so we still have a small race window?Quoted 2 lines
On the other hand, testing
I_CLEARhere probably would be safe anyway, and it'd be bonus points for consistency?So let's add the
I_CLEARtest?Quoted 3 lines
Same basic question for
generic_sync_sb_inodes, which has aBUG_ON(inode->i_state & I_FREEING), seems like this could checkI_CLWARas well?Yes, we can add
I_CLEARhere to catch more error condition.Thanks,
Fengguang---
skipI_CLEARstate inodes in writeback routinesThe
I_FREEINGtest in__sync_single_inode()is racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.Also extend the coverage of
BUG_ON(I_FREEING)toI_CLEAR.Reported-by Jeff Layton <[email protected]>Reported-by Eric Sandeen <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/fs-writeback.c __sync_single_inode(struct inode *inode, · line 321
Quoted 7 lines
fs/fs-writeback.c +1
-1@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {Is the whole if needed? I had an impression that everyone calling
__sync_single_inode()should better take care it does not race with inode freeing... SoWARN_ONwould be more appropriate IMHO.Quoted 5 lines
fs/fs-writeback.c +0
-0/* @@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ;Quoted 7 lines
fs/fs-writeback.c void generic_sync_sb_inodes(struct super · line 524
Quoted 5 lines
fs/fs-writeback.c +1
-1-BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);Looking at this code, it looks a bit suspicious. What prevents this
s_io
list scan to race with inode freeing? In particulargeneric_forget_inode()Good catch.
Quoted 3 lines
can drop
inode_lockto write the inode and in the mean timegeneric_sync_sb_inodes()can come, get a reference to the inode and start it's writeback... Subsequentiput()would then callgeneric_forget_inode()Another possibility:
generic_forget_inode
inode->i_state|=I_WILL_FREE;spin_unlock(&inode_lock); generic_sync_sb_inodes() spin_lock(&inode_lock); __iget(inode); __writeback_single_inode // see non zero i_count WARN_ON(inode->i_state & I_WILL_FREE);
Quoted 6 lines
Quoted 2 lines
I'm wondering why didn't we saw reports on the last
WARN_ON()? Did we missed something?I meant the above race in my description ;-). Anyway, the race can happen
only if we are unmounting the filesystem (normally, we bail out onsb->s_flags&MS_ACTIVEcheck - yes, it's a bit hidden and it also took me a while to understand why we weren't seeing tons of warnings...).
Ah OK. Just checked that all three callers of generic_sync_sb_inodes():
- writeback_inodes(): umount prevented
- pohmelfs_kill_super(): just before umount
- ubifs calls: too complex to be obvious..
At least the first two cases are safe, so we didn't see the error report ;)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
on the inode again. So shouldn't we skip I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW inodes in this scan like we do for later in the function for another scan?
Yes we should do this at least for safety. I_WILL_FREE means generic_forget_inode() is going to writeback the inode on its own, so generic_sync_sb_inodes() would better not to wade in.
Thanks,
Fengguang
PATCH 1/1 writeback: skip new or to-be-freed inodes
1) I_FREEING tests should be coupled with I_CLEAR
The two I_FREEING tests are racy because clear_inode() can set i_state to I_CLEAR between the clear of I_SYNC and the test of I_FREEING.
2) skip I_WILL_FREE inodes in generic_sync_sb_inodes() to avoid possible races with generic_forget_inode()
generic_forget_inode() sets I_WILL_FREE call writeback on its own, so generic_sync_sb_inodes() shall not try to step in and create possible races:
generic_forget_inode
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode_lock);
generic_sync_sb_inodes()
spin_lock(&inode_lock);
__iget(inode);
__writeback_single_inode
// see non zero i_count
may WARN here ==> WARN_ON(inode->i_state & I_WILL_FREE);
spin_unlock(&inode_lock); may call generic_forget_inode again ==> iput(inode);
The above race and warning didn't turn up because writeback_inodes() holds the s_umount lock, so generic_forget_inode() finds MS_ACTIVE and returns early. But we are not sure the UBIFS calls and future callers will guarantee that. So skip I_WILL_FREE inodes for the sake of safety.
--- fs/fs-writeback.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/fs-writeback.c +3 -3
@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/*
@@ -487,7 +487,7 @@ void generic_sync_sb_inodes(struct superbreak ; } -if (inode->i_state &I_NEW ) { +if (inode->i_state & (I_NEW |I_WILL_FREE )) {requeue_io (inode);continue ; }
@@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ; -BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);
Re: [PATCH] writeback: skip new or to-be-freed inodes
Quoted 4 lines · Wu Fengguang
The above race and warning didn't turn up because
writeback_inodes()holds thes_umountlock, sogeneric_forget_inode()findsMS_ACTIVEand returns early. But we are not sure the UBIFS calls and future callers will guarantee that. So skipI_WILL_FREEinodes for the sake of safety.
The inode states are a bit vague for me, but vs. UBIFS - feel free to ask questions.
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] writeback: skip new or to-be-freed inodes
Hi Artem,
Quoted 8 lines · Artem Bityutskiy
Quoted 4 lines · Wu Fengguang
The above race and warning didn't turn up because
writeback_inodes()holds thes_umountlock, sogeneric_forget_inode()findsMS_ACTIVEand returns early. But we are not sure the UBIFS calls and future callers will guarantee that. So skipI_WILL_FREEinodes for the sake of safety.The inode states are a bit vague for me, but vs. UBIFS - feel free to ask questions.
Thank you. Basically I'm not sure if UBIFS guarantees it won't be unmounted (hence the MS_ACTIVE bit is on) when calling generic_sync_sb_inodes() in shrink_liability() and ubifs_sync_fs().
Thanks,
Fengguang
PS: our previous discussions
> > Another possibility:
> >
> > generic_forget_inode
> > inode->i_state |= I_WILL_FREE;
> > spin_unlock(&inode_lock);
> > generic_sync_sb_inodes()
> > spin_lock(&inode_lock);
> > __iget(inode);
> > __writeback_single_inode
> > // see non zero i_count
> > WARN_ON(inode->i_state & I_WILL_FREE);
> >
> > I'm wondering why didn't we saw reports on the last WARN_ON()? > > Did we missed something?
> I meant the above race in my description ;-). Anyway, the race can happen
> only if we are unmounting the filesystem (normally, we bail out on > sb->s_flags & MS_ACTIVE check - yes, it's a bit hidden and it also took me > a while to understand why we weren't seeing tons of warnings...).
Ah OK. Just checked that all three callers of generic_sync_sb_inodes():
- writeback_inodes(): umount prevented
- pohmelfs_kill_super(): just before umount
- ubifs calls: too complex to be obvious..
At least the first two cases are safe, so we didn't see the error report ;)
Re: [PATCH] writeback: skip new or to-be-freed inodes
Quoted 3 lines · Wu Fengguang
Thank you. Basically I'm not sure if UBIFS guarantees it won't be unmounted (hence the
MS_ACTIVEbit is on) when callinggeneric_sync_sb_inodes()inshrink_liability()andubifs_sync_fs().
Btw, the call in ubifs_sync_fs should be superflous in the vfs-2.6#for-next tree. We now do make sure that all inodes are flushed before calling ->sync_fs with the wait parameter.
shrink_liability is a more interesting case, I don't understand enough of ubifs to comment on it.
Re: [PATCH] writeback: skip new or to-be-freed inodes
[CCed Adiran Hunter]
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Wu Fengguang
Thank you. Basically I'm not sure if UBIFS guarantees it won't be unmounted (hence the
MS_ACTIVEbit is on) when callinggeneric_sync_sb_inodes()inshrink_liability()andubifs_sync_fs().Btw, the call in
ubifs_sync_fsshould be superflous in the vfs-2.6#for-next tree. We now do make sure that all inodes are flushed before calling ->sync_fs with the wait parameter.
OK, thanks for letting know. Once this is merged upstream, I'll amend UBIFS.
Quoted 2 lines
shrink_liabilityis a more interesting case, I don't understand enough of ubifs to comment on it.
Well... I'm not sure if I can tell why we need this in few words. But I'll try.
UBIFS supports on-the-flight compression. This, and other factors lead to a situation when UBIFS does not know how much the dirty data in the page/inode caches will take on the flash. Indeed, how do we know how well the data will be compressed?
UBIFS has so called "budgeting" subsystem. This subsystem is responsible for accounting flash space. If user writes a new data page, which goes to the page cache and sits there, the budgeting sub-system decrements the free space counters by 4KiB. And so on.
At some point the free space counters in the budgeting subsystem reache zero, which means we do not have more space. However, in 99% of the cases this is not true, because the budgeting subsystem's calculations are _very_ pessimistic, they always assume the worst case scenario like the data are uncompressible.
So consider a situation when user writes a new data page. First of all, the ->write_begin function will call a budgeting sub-system function in order to reserve flash space for this new data page.
The budgeting subsystem will see that space counters are zero. And what it will do it will call the 'shrink_liability()' function, which, among other things, may call the 'generic_sync_sb_inodes()' function, which will force write-back, and this will give us some space. Indeed, when we actually write the data back, we'll see how much flash space they really take. And in 99% of cases they will take less than we budgeted for, usually much less.
This is the rough idea. In practice things are more complex, and there are factors like inability to know how much of dirty space may be reclaimed, what will be the index size after commit, etc. This all makes the budgeting subsystem complex and difficult to understand. Moreover, we still consider it as a work in progress, because we use too rough calculations, and there are too many heuristics.
Here you may read some more information about UBIFS flash accounting issues:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_spaceacc
You may also find a lot of info here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_documentation
Especially in this document:
http://www.linux-mtd.infradead.org/doc/ubifs_whitepaper.pdf
but it is not easy reading. You may search for "budget" in the doc.
HTH.
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] writeback: skip new or to-be-freed inodes
In reply to Wu FengguangQuoted 8 lines · Wu Fengguang
Hi Artem,
Quoted 5 lines · Artem Bityutskiy
Quoted 4 lines · Wu Fengguang
The above race and warning didn't turn up because
writeback_inodes()holds thes_umountlock, sogeneric_forget_inode()findsMS_ACTIVEand returns early. But we are not sure the UBIFS calls and future callers will guarantee that. So skipI_WILL_FREEinodes for the sake of safety.
Quoted 6 lines
Quoted 2 lines
The inode states are a bit vague for me, but vs. UBIFS - feel free to ask questions.
Thank you. Basically I'm not sure if UBIFS guarantees it won't be unmounted (hence the
MS_ACTIVEbit is on) when callinggeneric_sync_sb_inodes()inshrink_liability()andubifs_sync_fs().
To be frank my VFS knowledge is not good enough to give you a good answer. MS_ACTIVE seems to be set by file-systems when they are mounted, and cleaned by VFS when unmounting. I guess MS_ACTIVE is used by FS-es to check whether unmounting is in progress or not. Anyway, UBIFS does not use it.
The generic_sync_sb_inodes() is called only from within VFS operations, e.g., from ->create, ->rename, ->mknod, ->write_begin, ->setattr, etc. I mean, it is called from UBIFS implementations of the above calls. UBIFS never calls generic_sync_sb_inodes() function by itself, e.g., from the UBIFS background thread.
Also, all calls to generic_sync_sb_inodes() are from within VFS operations which need writing, e.g., VFS called 'mnt_want_write()' for all of them.
I think VFS must protect the FS from being unmunted while it is in the middle of an operation, right? I'm just not sure how this mechanism works. This would mean that yes, we cannot be unmounted while we are in generic_sync_sb_inodes() called by UBIFS.
Did this help :-) ?
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Jan Kara
maintainerRe: [PATCH] writeback: skip new or to-be-freed inodes
In reply to Wu FengguangQuoted 21 lines · Wu Fengguang
1)
I_FREEINGtests should be coupled withI_CLEARThe two
I_FREEINGtests are racy becauseclear_inode()can seti_statetoI_CLEARbetween the clear ofI_SYNCand the test ofI_FREEING.2) skip
I_WILL_FREEinodes ingeneric_sync_sb_inodes()to avoid possible races withgeneric_forget_inode()
generic_forget_inode()setsI_WILL_FREEcall writeback on its own, sogeneric_sync_sb_inodes()shall not try to step in and create possible races:
generic_forget_inodeinode->i_state |= I_WILL_FREE; spin_unlock(&inode_lock); generic_sync_sb_inodes() spin_lock(&inode_lock); __iget(inode); __writeback_single_inode // see non zero i_count may WARN here ==> WARN_ON(inode->i_state & I_WILL_FREE); spin_unlock(&inode_lock);
Quoted 6 lines
may call
generic_forget_inodeagain ==> iput(inode);The above race and warning didn't turn up because
writeback_inodes()holds thes_umountlock, sogeneric_forget_inode()findsMS_ACTIVEand returns early. But we are not sure the UBIFS calls and future callers will guarantee that. So skipI_WILL_FREEinodes for the sake of safety.
OK, the patch looks fine. Acked-by: Jan Kara <[email protected]>
Honza
Quoted 38 lines
CC Jan Kara <[email protected]>CC Eric Sandeen <[email protected]>Acked-by Jeff Layton <[email protected]>CC Masayoshi MIZUMA <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/fs-writeback.c +3
-3@@ -316,7 +316,7 @@ __sync_single_inode(struct inode *inode,spin_lock (&inode_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_SYNC ; -if (!(inode->i_state &I_FREEING )) { +if (!(inode->i_state & (I_FREEING |I_CLEAR ))) {if (!(inode->i_state &I_DIRTY ) &&mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/* @@ -487,7 +487,7 @@ void generic_sync_sb_inodes(struct superbreak ; } -if (inode->i_state &I_NEW ) { +if (inode->i_state & (I_NEW |I_WILL_FREE )) {requeue_io (inode);continue ; }@@ -518,7 +518,7 @@ void generic_sync_sb_inodes(struct superif (current_is_pdflush () && !writeback_acquire (bdi))break ; -BUG_ON (inode->i_state &I_FREEING ); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped;__writeback_single_inode (inode, wbc);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR