fs: clear_inode failed with nrpages not zero!
Hi all:
I am running a redhat 2.6.32-279 offical kernel. Under heavy work
load and memory pressure, in my case, running ltp test for about 20 hours, kernel oops happened. Say concretely, a testcase process open a file, truncate to 128M, mmap, munmap and close the file, this circle repeatedly when kernel hangs. Through the vmcore, I also find it hangs at: BUG_ON(inode->i_data.nrpages) in function clear_inode, which means the truncate_inode_pages faild to decrase nrpages to 0. I have google this problem and find no clear solutions but make me confused. The comment of function truncate_inode_pages says that after it return, the nrpages may not be zero.
My understanding is: the page reclaime migth still in the process
of deletion of the page. Jan Kara once post a patch, which use spin_lock to sync the radix tree and nrpages. This kernel already contains this patch. Then problem come: When kernel hangs, the nrpages is not a small number like 1 or 2, but a bigger one, more than 500 or 700! So I think even we take some sync measures before clear inode, the function truncate_inode_pages together with other reclaim functions failed to set nrpages to zero. By dump the vmcore, I also find the radix tree is also not empty but with some slots left.
Then I think:
1. The fault might happen at pagevec_lookup, which return no page
even the radix tree is in fact not empty. Because lookup uses the rcu lock, is it possible a race condition
happened in the lookup process and lead the function return
unexpectedly? If possiable, how dose it happened ?
2. I find Johannes Weiner post a
patch(http://www.spinics.net/lists/linux-fsdevel/msg72395.html), which has following code:
+ if (nrpages || nrshadows) {
+ /*
+ * As truncation uses a lockless tree lookup, cycle
+ * the tree lock to make sure any ongoing tree
+ * modification that does not see AS_EXITING is
+ * completed before starting the final truncate.
+ */
+ spin_lock_irq(&mapping->tree_lock);
+ spin_unlock_irq(&mapping->tree_lock);
+
+ truncate_inode_pages(mapping, 0);
+ }
which wrapped the truncate_inode_pages in functiontruncate_inode_pages_final. Does it make sence to my problem ?
Any suggestion will be appreciated!
Jan Kara
reviewerHello,
Quoted 1 line · hitmoon
I am running a redhat 2.6.32-279 offical kernel.
Well, in that case you should consider contacting RH support instead of
general community forum... Also 2.6.32 is pretty old and RH (similarly as other enterprise distributors) has lots of stuff added on top of it. So it is hard to help you.
Quoted 45 lines
Under heavy work load and memory pressure, in my case, running ltp test for about 20 hours, kernel oops happened. Say concretely, a testcase process open a file, truncate to 128M, mmap, munmap and close the file, this circle repeatedly when kernel hangs. Through the vmcore, I also find it hangs at:
BUG_ON(inode->i_data.nrpages)in functionclear_inode, which means thetruncate_inode_pagesfaild to decrase nrpages to 0. I have google this problem and find no clear solutions but make me confused. The comment of functiontruncate_inode_pagessays that after it return, the nrpages may not be zero.My understanding is: the page reclaime migth still in the
process of deletion of the page. Jan Kara once post a patch, which usespin_lockto sync the radix tree and nrpages. This kernel already contains this patch. Then problem come: When kernel hangs, the nrpages is not a small number like 1 or 2, but a bigger one, more than 500 or 700! So I think even we take some sync measures before clear inode, the functiontruncate_inode_pagestogether with other reclaim functions failed to set nrpages to zero. By dump the vmcore, I also find the radix tree is also not empty but with some slots left.Then I think:
1. The fault might happen atpagevec_lookup, which return no
page even the radix tree is in fact not empty. Because lookup uses the rcu lock, is it possible a race condition
happened in the lookup process and lead the function return
unexpectedly? If possiable, how dose it happened ?
2. I find Johannes Weiner post a
patch(http://www.spinics.net/lists/linux-fsdevel/msg72395.html), which has following code:(quoted diff) +12
-0+ if (nrpages || nrshadows) { + /* + * As truncation uses a lockless tree lookup, cycle + * the tree lock to make sure any ongoing tree + * modification that does not see AS_EXITING is + * completed before starting the final truncate. + */ + spin_lock_irq(&mapping->tree_lock); + spin_unlock_irq(&mapping->tree_lock); + + truncate_inode_pages(mapping, 0); + } which wrapped the truncate_inode_pages in function
Quoted 1 line
truncate_inode_pages_final. Does it make sence to my problem ?
This shouldn't be really related. That is specific to Johannes' patch set
adding new special radix tree entries.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR