oops in 2.4.25 prune_icache() called from kswapd
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Shame the RH bugzilla (#155289) requires super priviledges to be accessed.
I've got around reading Albert's description of the race - thanks BTW. (its attached below for reference)
It seems to me that only window open in mainline is between iput() and prune_icache(), while iput() sleeps on sync_one() with the inode being:
- on the unused list
- and with i_count set to zero
prune_icache() is free to invalidate and destroy the inode in the meantime, causing iput()'s sync_one() to __refile_inode() the NULL entry to the unused list later on.
If that is indeed the case, removing __refile_inode() from the nonzero inode->i_nlink path should close that window.
Chris, can you please test the attached "iput-iprune-race.patch" with your usual irqbalance enable environment ?
Quoted 11 lines · Albert Chu
Howdy everyone,
Quoted 2 lines
Albert Chu, CC'ed, has suggested the below as a fix. Albert, any new info on this, or have these two patches cleared up the problem well?
The
__refile_inode()patch below fixed the problem for us on our clusters (running RHEL3). Theclear_inode()patch is something Redhat (I think Larry Woodman) gave to us to fix a problem he believes is in the same general area. We haven't had any additional problems adding his second patch.
Quoted 4 lines
Quoted 2 lines
BTW, the below mail says that these are workarounds and a real fix is on the way. Has that been rolled in as well?
Sorry, not too sure. :-(
Albert's description:
Quoted 38 lines
Just thought I'd let you know what's up. We think we're close to getting to the bottom of this. We think the race is between
iput()and__refile_inode(). The Redhat kernel is of course different than the mainline kernel, but I think the same bug exists in the mainline. The example below illustrates the race betweeniput()and__sync_one(), but it could occur with other areas that all__refile_inode(). For us, I think we're hitting it in__sync_one()andprune_icache(). (Theprune_icache()call to__refile_inode()doesn't seem to be in the mainline though).proc 0:
callsiput()and locksinode_lock.
iput removes the inode off of thei_list
unlocksinode_lock(proc 1 will now at some point grabinode_lock) callsclear_inode()
(this is key) gets past the call towait_on_inode();
at this pointclear_inode()and the remainder ofiput()does not care aboutI_LOCKorinode_lock.proc 1:
calls__sync_one()withinode_lock.
setsI_LOCK
do stuff before__refile_inode()is called, all I_LOCK/inode_lock stuff doesn't matter.proc 0:
setsi_state=I_CLEAR
iput callsdestroy_inode()proc 1:
calls__refile_inodeand we have ourselves a corrupted inode on the
inode_unusedlist.I'm not sure if you can see it or not, but Redhat bugzilla 155289 is tracking this.
Al
Hi,
Marcello, your patch didn't come through. Unfortunately I can't test it anymore since my environment has now changed, but hopefully Albert or Prakash can try it when you resend.
Thanks,
Chris
Quoted 83 lines · Marcelo Tosatti
Hi,
Shame the RH bugzilla (#155289) requires super priviledges to be accessed.
I've got around reading Albert's description of the race - thanks BTW. (its attached below for reference)
It seems to me that only window open in mainline is between
iput()andprune_icache(), whileiput()sleeps onsync_one()with the inode being:- on the unused list
- and withi_countset to zero
prune_icache()is free to invalidate and destroy the inode in the meantime, causing iput()'ssync_one()to__refile_inode()the NULL entry to the unused list later on.If that is indeed the case, removing
__refile_inode()from the nonzeroinode->i_nlinkpath should close that window.Chris, can you please test the attached "iput-iprune-race.patch" with your usual irqbalance enable environment ?
Quoted 11 lines · Albert Chu
Howdy everyone,
Quoted 2 lines
Albert Chu, CC'ed, has suggested the below as a fix. Albert, any new info on this, or have these two patches cleared up the problem well?
The
__refile_inode()patch below fixed the problem for us on our clusters (running RHEL3). Theclear_inode()patch is something Redhat (I think Larry Woodman) gave to us to fix a problem he believes is in the same general area. We haven't had any additional problems adding his second patch.Quoted 4 lines
Quoted 2 lines
BTW, the below mail says that these are workarounds and a real fix is on the way. Has that been rolled in as well?
Sorry, not too sure. :-(
Albert's description:
Quoted 38 lines
Just thought I'd let you know what's up. We think we're close to getting to the bottom of this. We think the race is between
iput()and__refile_inode(). The Redhat kernel is of course different than the mainline kernel, but I think the same bug exists in the mainline. The example below illustrates the race betweeniput()and__sync_one(), but it could occur with other areas that all__refile_inode(). For us, I think we're hitting it in__sync_one()andprune_icache(). (Theprune_icache()call to__refile_inode()doesn't seem to be in the mainline though).proc 0:
callsiput()and locksinode_lock.
iput removes the inode off of thei_list
unlocksinode_lock(proc 1 will now at some point grabinode_lock) callsclear_inode()
(this is key) gets past the call towait_on_inode();
at this pointclear_inode()and the remainder ofiput()does not care aboutI_LOCKorinode_lock.proc 1:
calls__sync_one()withinode_lock.
setsI_LOCK
do stuff before__refile_inode()is called, all I_LOCK/inode_lock stuff doesn't matter.proc 0:
setsi_state=I_CLEAR
iput callsdestroy_inode()proc 1:
calls__refile_inodeand we have ourselves a corrupted inode on the
inode_unusedlist.I'm not sure if you can see it or not, but Redhat bugzilla 155289 is tracking this.
Al
Quoted 5 lines · Chris Caputo
Hi,
Marcello, your patch didn't come through. Unfortunately I can't test it anymore since my environment has now changed, but hopefully Albert or Prakash can try it when you resend.
Albert, Prakash, any of you using stock v2.4?
Here's the diff:
fs/inode.c renamed from fs/inode.c.orig +9 -6
@@ -1238,15 +1238,18 @@BUG (); }else {if (!list_empty (&inode->i_hash)) { -if (!(inode->i_state & (I_DIRTY |I_LOCK ))) -__refile_inode (inode); - inodes_stat.nr_unused++; -spin_unlock (&inode_lock); -if (!sb || (sb->s_flags &MS_ACTIVE )) +if (!sb || (sb->s_flags &MS_ACTIVE )) { +if (!(inode->i_state & + (I_DIRTY |I_LOCK ))) { +__refile_inode (inode); + inodes_stat.nr_unused++; + } +spin_unlock (&inode_lock);return ; + } +spin_unlock (&inode_lock);write_inode_now (inode,1 );spin_lock (&inode_lock); - inodes_stat.nr_unused--;list_del_init (&inode->i_hash); }list_del_init (&inode->i_list);
Quoted 88 lines
Thanks,
ChrisQuoted 83 lines · Marcelo Tosatti
Hi,
Shame the RH bugzilla (#155289) requires super priviledges to be accessed.
I've got around reading Albert's description of the race - thanks BTW. (its attached below for reference)
It seems to me that only window open in mainline is between
iput()andprune_icache(), whileiput()sleeps onsync_one()with the inode being:- on the unused list
- and withi_countset to zero
prune_icache()is free to invalidate and destroy the inode in the meantime, causing iput()'ssync_one()to__refile_inode()the NULL entry to the unused list later on.If that is indeed the case, removing
__refile_inode()from the nonzeroinode->i_nlinkpath should close that window.Chris, can you please test the attached "iput-iprune-race.patch" with your usual irqbalance enable environment ?
Quoted 11 lines · Albert Chu
Howdy everyone,
Quoted 2 lines
Albert Chu, CC'ed, has suggested the below as a fix. Albert, any new info on this, or have these two patches cleared up the problem well?
The
__refile_inode()patch below fixed the problem for us on our clusters (running RHEL3). Theclear_inode()patch is something Redhat (I think Larry Woodman) gave to us to fix a problem he believes is in the same general area. We haven't had any additional problems adding his second patch.Quoted 4 lines
Quoted 2 lines
BTW, the below mail says that these are workarounds and a real fix is on the way. Has that been rolled in as well?
Sorry, not too sure. :-(
Albert's description:
Quoted 38 lines
Just thought I'd let you know what's up. We think we're close to getting to the bottom of this. We think the race is between
iput()and__refile_inode(). The Redhat kernel is of course different than the mainline kernel, but I think the same bug exists in the mainline. The example below illustrates the race betweeniput()and__sync_one(), but it could occur with other areas that all__refile_inode(). For us, I think we're hitting it in__sync_one()andprune_icache(). (Theprune_icache()call to__refile_inode()doesn't seem to be in the mainline though).proc 0:
callsiput()and locksinode_lock.
iput removes the inode off of thei_list
unlocksinode_lock(proc 1 will now at some point grabinode_lock) callsclear_inode()
(this is key) gets past the call towait_on_inode();
at this pointclear_inode()and the remainder ofiput()does not care aboutI_LOCKorinode_lock.proc 1:
calls__sync_one()withinode_lock.
setsI_LOCK
do stuff before__refile_inode()is called, all I_LOCK/inode_lock stuff doesn't matter.proc 0:
setsi_state=I_CLEAR
iput callsdestroy_inode()proc 1:
calls__refile_inodeand we have ourselves a corrupted inode on the
inode_unusedlist.I'm not sure if you can see it or not, but Redhat bugzilla 155289 is tracking this.
Al