oops in 2.4.25 prune_icache() called from kswapd
Quoted 1 line
Quoted 1 line
Albert, Prakash, any of you using stock v2.4?
Sorry, we don't use a stock kernel in our environment :-(
Quoted 1 line
Do you have any inputs on reproducing this?
Sorry. We were a bit resource fortunate when debugging this. Heavy I/O scientific simulations on a 1000 node cluster got us about 5-10 hits a day.
Al
--
Albert Chu
[email protected]
Lawrence Livermore National Laboratory
----- Original Message -----
From: Chris Caputo <[email protected]>
Date: Sunday, June 19, 2005 4:07 pm
Subject: RE: RE: oops in 2.4.25 prune_icache() called from kswapd
Quoted 193 lines
My basic repro method was:
--
0) start irqbalance
1) runloop_dbench, which is the following dbench script which uses client_plain.txt:#!/bin/sh while [ 1 ] do date dbench 22) wait for oops
--I think I was using dbench-2.1:
http://samba.org/ftp/tridge/dbench/dbench-2.1.tar.gz
In my case irqbalance was key. If I didn't run it I never got the problem. I think irqbalance just did a good job of exasperating a race
condition in some way.Chris
Quoted 3 lines · Prakash
Hi Marcelo,
We are using 2.4.20 and would be able test it. But before that,
we are
Quoted 14 lines
trying to recreate the problem. We just had one occurrence of this problem. Do you have any inputs on reproducing this?
Thanks,
Prakash-----Original Message-----
From: Marcelo Tosatti [mailto:[email protected]]
Sent: Saturday, June 18, 2005 3:34 PM
To: Chris Caputo
Cc: Albert Chu; David Woodhouse; Al Viro; Velupula, Prakash; [email protected]; [email protected]
Subject: Re: RE: oops in 2.4.25prune_icache()called from kswapdQuoted 4 lines
Quoted 3 lines · Chris Caputo
Hi,
Marcello, your patch didn't come through. Unfortunately I can't
test>> it anymore since my environment has now changed, but hopefully Albert
Quoted 38 lines
Quoted 1 line
or Prakash can try it when you resend.
Albert, Prakash, any of you using stock v2.4?
Here's the diff:
fs/inode.c renamed from fs/inode.c.orig +9
-6@@ -1238,15 +1238,18 @@BUG (); }else {if (!list_empty (&inode->i_hash)) { -if (!(inode->i_state & (I_DIRTY|I_LOCK))) -__refile_inode (inode); - inodes_stat.nr_unused++; -spin_unlock (&inode_lock); -if (!sb || (sb->s_flags &MS_ACTIVE )) +if (!sb || (sb->s_flags &MS_ACTIVE )) { +if (!(inode->i_state & + (I_DIRTY |I_LOCK ))) { +__refile_inode (inode); + inodes_stat.nr_unused++; + } +spin_unlock (&inode_lock);return ; + } +spin_unlock (&inode_lock);write_inode_now (inode,1 );spin_lock (&inode_lock); - inodes_stat.nr_unused--;list_del_init (&inode->i_hash); }list_del_init(&inode->i_list);Quoted 4 lines
Thanks,
ChrisQuoted 12 lines
Quoted 7 lines · Marcelo Tosatti
Quoted 6 lines
Hi,
Shame the RH bugzilla (#155289) requires super priviledges to be accessed.
I've got around reading Albert's description of the race - thanks
BTW.
Quoted 4 lines
Quoted 4 lines
(its attached below for reference)
It seems to me that only window open in mainline is between
iput()andprune_icache(), whileiput()sleeps onsync_one()with theinode>>> being:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
- on the unused list
- and withi_countset to zero
prune_icache()is free to invalidate and destroy the inode in the meantime, causing iput()'ssync_one()to__refile_inode()the NULL entry to the unused list later on.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If that is indeed the case, removing
__refile_inode()from the nonzeroinode->i_nlinkpath should close that window.Chris, can you please test the attached "iput-iprune-
race.patch" with
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
your usual irqbalance enable environment ?
Quoted 3 lines · Albert Chu
Howdy everyone,
Quoted 1 line
Albert Chu, CC'ed, has suggested the below as a fix. Albert,
any>>>>> new info on this, or have these two patches cleared up the problem> well?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The
__refile_inode()patch below fixed the problem for us on our clusters (running RHEL3). Theclear_inode()patch is something Redhat (I think Larry Woodman) gave to us to fix a problem he believes is in the same general area. We haven't had anyadditional>
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
problems adding his second patch.
Quoted 1 line
BTW, the below mail says that these are workarounds and a
real fix
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
is on the way. Has that been rolled in as well?
Sorry, not too sure. :-(
Albert's description:
Quoted 2 lines
Just thought I'd let you know what's up. We think we're close to getting to the bottom of this. We think the race is between
iput()>>>> and__refile_inode(). The Redhat kernel is of course different thanQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the mainline kernel, but I think the same bug exists in the mainline. The example below illustrates the race between
iput()andQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
__sync_one(), but it could occur with other areas that all__refile_inode(). For us, I think we're hitting it in
__sync_one()>>>> andprune_icache(). (TheQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
prune_icache()call to__refile_inode()doesn't seem to be in the mainline though).proc 0:
callsiput()and locksinode_lock.
iput removes the inode off of thei_listunlocksinode_lock(proc 1
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will now at some point grab
inode_lock) callsclear_inode()(this is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
key) gets past the call to
wait_on_inode(); at this pointclear_inode()and the remainder ofiput()does not care about
I_LOCK>Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
or
inode_lock.proc 1:
calls__sync_one()withinode_lock.
setsI_LOCK
do stuff before__refile_inode()is called, all I_LOCK/inode_lock stuff doesn't matter.proc 0:
setsi_state=I_CLEAR
iput callsdestroy_inode()Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
proc 1:
calls__refile_inodeand we have ourselves a corrupted inode on the
inode_unusedlist.I'm not sure if you can see it or not, but Redhat bugzilla
155289 is
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
tracking this.
Al