sunrpc.ko: RPC cache fix races
Subsystems (2)
Quoted 100 lines · On 20 Feb 2013 14:57:07 +0100 [email protected] wrote:
Quoted 98 lines · On 20 Feb 2013 04:09:00 +0100 [email protected] wrote:
Quoted 40 lines · On 19 Feb 2013 18:08:40 +0100 [email protected] wrote:
Second attempt using the correct FROM. Sorry for the noise.
Hi,
I found a problem in sunrpc.ko on a SLES11 SP1 (2.6.32.59-0,7.1) and fixed it (see first patch ifor 2.6.32.60 below).
For us the patch works fine (tested on 2.6.32.59-0.7.1).AFAICS from the code, the problem seems to persist in current kernel versions also. Thus, I added the second patch for 3.7.9. As the setup to reproduce the problem is quite complex, I couldn't test the second patch yet. So consider this one as a RFC.
Best regards,
BodoPlease CC me, I'm not on the list.
=========================================
From: Bodo Stroesser <[email protected]>
Date: Fri, 08 Feb 2013
Subject: [PATCH] net: sunrpc: fix races in RPC cacheWe found the problem and tested the patch on a SLES11 SP1 2.6.32.59-0.7.1
This patch applies to linux-2.6.32.60 (changed
monotonic_seconds-->get_seconds())Sporadically NFS3 RPC requests to the nfs server are dropped due to
cache_check()(net/sunrpc/cache.c) returning-ETIMEDOUTfor an entry of the "auth.unix.gid" cache.
In this case, no NFS reply is sent to the client.The reason for the dropped requests are races in
cache_check()whencache_make_upcall()returns-EINVAL(because it is called for a cache without readers) andcache_check()therefore refreshes the cache entry (rv ==-EAGAIN).Quoted 4 lines
There are three details that need to be changed:
1)cache_revisit_request()must not be called beforecache_fresh_locked()has updated the cache entry, ascache_revisit_request()wakes up threads waiting for the cache entry to be updated.This certainly seems correct. It is wrong to call
cache_revisit_request()so early.Quoted 2 lines
The explicit call to
cache_revisit_request()is not needed, ascache_fresh_unlocked()calls it anyway.But
cache_fresh_unlockedis only called if "rv ==-EAGAIN", however we also need to call it in the case where "age > refresh_age/2" - it must always be called after clearingCACHE_PENDING.Also,
cache_fresh_unlocked()only callscache_revisit_request()ifCACHE_PENDINGis set, but we have just cleared it! Some definitely something wrong here. (Note that I'm looking at the SLES 2.6.32 code at the moment, mainline is a bit different).Quoted 2 lines
(But in case of not updating the cache entry,
cache_revisit_request()must be called. Thus, we use a fall through in the "case").Hmm... I don't like case fallthroughs unless they have nice big comments:
/* FALLTHROUGH */or similar. :-)
Quoted 3 lines
2)
CACHE_PENDINGmust not be cleared beforecache_fresh_locked()has updated the cache entry, ascache_defer_req()can return without really sleeping if it detectsCACHE_PENDINGunset.Agreed. So we should leave the clearing of
CACHE_PENDINGtocache_fresh_unlocked().Quoted 8 lines
(In case of not updating the cache entry again we use the fall
through)
3) Imagine a thread that callscache_check()and gets rv =-ENOENTfromcache_is_valid(). Then it setsCACHE_PENDINGsand callscache_make_upcall().
We assume that meanwhileget_seconds()advances to the next sec. and a second thread also callscache_check(). It gets-EAGAINfromcache_is_valid()for the same cache entry. AsCACHE_PENDINGstill isQuoted 8 lines
set, it calls
cache_defer_req()immediately and waits for a wakeup from the first thread.
Aftercache_make_upcall()returned-EINVAL, the first thread does not update the cache entry as it had got rv =-ENOENT, but wakes up the second thread by callingcache_revisit_request().
Thread two wakes up, callscache_is_valid()and again gets-EAGAIN. Thus, the result of the secondcache_check()is-ETIMEDOUTand the NFS request is dropped.Yep, that's not so good....
Quoted 5 lines
To solve this, the cache entry now is updated not only if rv ==
-EAGAINbut if rv ==-ENOENTalso. This is a sufficient workaround, as the first thread would have to stay incache_check()between its call tocache_is_valid()and clearingCACHE_PENDINGfor more than 60 seconds to break the workaround.
Quoted 4 lines
Quoted 2 lines
Still, it isn't nice to just have a work-around. It would be best to have a fix.
The key problem here is thatcache_is_valid()is time-sensitive. This have been address in mainline -cache_is_validdoesn't depend on the current time there.So, the solution would be a backport of the current mainline code ...
That would always be my preference, when it is practical.
Quoted 4 lines
Anyway, I think for SLES11 SP1 and 2.6.32.60 the work-around would be sufficient.
BTW: it has the positive side effect, that - while a cache entry is in its second half of life - no longer eachcache_check()tries to do acache_make_upcall().
I agree that is an improvement.
Quoted 45 lines
Quoted 43 lines
Quoted 19 lines
Signed-off-by Bodo Stroesser <[email protected]>---
net/sunrpc/cache.c +1
-3@@ -233,15 +233,14 @@ int cache_check(struct cache_detail *detif (!test_and_set_bit (CACHE_PENDING , &h->flags)) {switch (cache_make_upcall (detail, h)) {case -EINVAL : -clear_bit (CACHE_PENDING , &h->flags); -cache_revisit_request (h); -if (rv == -EAGAIN ) { +if (rv == -EAGAIN || rv == -ENOENT ) {set_bit (CACHE_NEGATIVE , &h->flags);net/sunrpc/cache.c int cache_check(struct cache_detail *det · line 246
Quoted 10 lines
net/sunrpc/cache.c +2
-1cache_fresh_locked (h,get_seconds ()+CACHE_NEW_EXPIRY ); +clear_bit (CACHE_PENDING , &h->flags);cache_fresh_unlocked (h, detail); rv = -ENOENT ; +break ; } -break ;case -EAGAIN :clear_bit (CACHE_PENDING , &h->flags);I agree with some of this....
Maybe:
switch(cache_make_upcall(detail, h)){
case-EINVAL:if (rv) { set_bit(CACHE_NEGATIVE, &h->flags); cache_fresh_locked(h, get_seconds() + CACHE_NEW_EXPIRY); rv = -ENOENT; } /* FALLTHROUGH */ case -EAGAIN: cache_fresh_unlocked(h, detail);Quoted 1 line
}
Quoted 5 lines
I agree, your patch is obviously better than the mine. But let me suggest one little change: I would like to substitute
cache_fresh_unlocked()byclear_bit()andcache_revisit_request(), as the call tocache_dequeue()incache_fresh_unlocked()seems to be obsolete here:
It is exactly this sort of thinking (on my part) that got us into this mess in the first place. I reasoned that the full locking/testing/whatever wasn't necessary and took a short cut. It wasn't a good idea.
Given that this is obviously difficult code to get right, we should make it as easy to review as possible. Have "cache_fresh_unlocked" makes it more obviously correct, and that is a good thing.
Maybe cache_dequeue isn't needed here, but it won't hurt so I'd much rather have the clearer code.
In fact, I'd also like to change
if (test_and_clear_bit(CACHE_PENDING, &ch->flags)) cache_dequeue(current_detail, ch); cache_revisit_request(ch);
near the end of cache_clean to call cache_fresh_unlocked().
NeilBrown