sunrpc/cache.c: races while updating cache entries
Subsystems (2)
T24gMDUgQXByIDIwMTMgMjM6MDk6MDAgKzAxMDAgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRz QGZpZWxkc2VzLm9yZz4gd3JvdGU6Cj4gT24gRnJpLCBBcHIgMDUsIDIwMTMgYXQgMDU6MzM6 NDlQTSArMDIwMCwgQm9kbyBTdHJvZXNzZXIgd3JvdGU6Cj4gPiBPbiAwNSBBcHIgMjAxMyAx NDo0MDowMCArMDEwMCBKLiBCcnVjZSBGaWVsZHMgPGJmaWVsZHNAZmllbGRzZXMub3JnPiB3 cm90ZToKPiA+ID4gT24gVGh1LCBBcHIgMDQsIDIwMTMgYXQgMDc6NTk6MzVQTSArMDIwMCwg Qm9kbyBTdHJvZXNzZXIgd3JvdGU6Cj4gPiA+ID4gVGhlcmUgaXMgbm8gcmVhc29uIGZvciBh cG9sb2dpZXMuIFRoZSB0aHJlYWQgbWVhbndoaWxlIHNlZW1zIHRvIGJlIGEgYml0Cj4gPiA+ ID4gY29uZnVzaW5nIDotKQo+ID4gPiA+IAo+ID4gPiA+IEN1cnJlbnQgc3RhdGUgaXM6Cj4g PiA+ID4gCj4gPiA+ID4gLSBOZWlsIEJyb3duIGhhcyBjcmVhdGVkIHR3byBzZXJpZXMgb2Yg cGF0Y2hlcy4gT25lIGZvciBTTEVTMTEtU1AxIGFuZCBhCj4gPiA+ID4gICBzZWNvbmQgb25l IGZvciAtU1AyCj4gPiA+ID4gCj4gPiA+ID4gLSBBRkFJQ1MsIHRoZSBzZXJpZXMgZm9yIC1T UDIgd2lsbCBtYXRjaCB3aXRoIG1haW5saW5lIGFsc28uCj4gPiA+ID4gCj4gPiA+ID4gLSBU b2RheSBJIGZvdW5kIGFuZCBmaXhlZCB0aGUgKGhvcGVmdWxseSkgbGFzdCBwcm9ibGVtIGlu IHRoZSAtU1AxIHNlcmllcy4KPiA+ID4gPiAgIE15IHRlc3QgdXNpbmcgdGhpcyBwYXRjaHNl dCB3aWxsIHJ1biB1bnRpbCBNb25kYXkuCj4gPiA+ID4gCj4gPiA+ID4gLSBQcm92aWRlZCB0 aGUgdGVzdCBvbiBTUDEgc3VjY2VlZHMsIHByb2JhYmx5IG9uIFR1ZXNkYXkgSSdsbCBzdGFy dCB0byB0ZXN0Cj4gPiA+ID4gICB0aGUgcGF0Y2hlcyBmb3IgU1AyIChhbmQgbWFpbmxpbmUp LiBJZiBpdCBydW5zIGZpbmUsIHdlJ2xsIGhhdmUgYSB0ZXN0ZWQKPiA+ID4gPiAgIHBhdGNo c2V0IG5vdCBsYXRlciB0aGFuIE1vbiAxNXRoLgo+ID4gPiAKPiA+ID4gT0ssIGdyZWF0LCBh cyBsb25nIGFzIGl0IGhhc24ndCBqdXN0IGJlZW4gZm9yZ290dGVuIQo+ID4gPiAKPiA+ID4g SSdkIGFsc28gYmUgY3VyaW91cyB0byB1bmRlcnN0YW5kIHdoeSB3ZSBhcmVuJ3QgZ2V0dGlu ZyBhIGxvdCBvZgo+ID4gPiBjb21wbGFpbnRzIGFib3V0IHRoaXMgZnJvbSBlbHNld2hlcmUu Li4uICBJcyB0aGVyZSBzb21ldGhpbmcgdW5pcXVlCj4gPiA+IGFib3V0IHlvdXIgc2V0dXA/ ICBEbyB0aGUgYnVncyB0aGF0IHJlbWFpbiB1cHN0cmVhbSB0YWtlIGEgbG9uZyB0aW1lIHRv Cj4gPiA+IHJlcHJvZHVjZT8KPiA+ID4gCj4gPiA+IC0tYi4KPiA+ID4gCj4gPiAKPiA+IEl0 J3Mgbm8gc2VjcmV0LCB3aGF0IHdlIGFyZSBkb2luZy4gU28gbGV0IG1lIHRyeSB0byBleHBs YWluOgo+IAo+IFRoYW5rcyBmb3IgdGhlIGRldGFpbGVkIGV4cGxhbmF0aW9uISAgSSdsbCBs b29rIGZvcndhcmQgdG8gdGhlIHBhdGNoZXMuCj4gCj4gLS1iLgo+IAoKTGV0IG1lIGdpdmUg YW4gaW50ZXJtZWRpYXRlIHJlc3VsdDoKClRoZSB0ZXN0IG9mIHRoZSAtU1AxIHBhdGNoIHNl cmllcyBzdWNjZWVkZWQuCgpXZSBzdGFydGVkIHRoZSB0ZXN0IG9mIHRoZSAtU1AyIChhbmQg bWFpbmxpbmUpIHNlcmllcyBvbiBUdWUsIDl0aCwgYnV0IGhhZCBubwpzdWNjZXNzLgpXZSBk aWQgX25vdF8gZmluZCBhIHByb2JsZW0gd2l0aCB0aGUgcGF0Y2hlcywgYnV0IHVuZGVyIC1T UDIgb3VyIHRlc3Qgc2NlbmFyaW8KaGFzIGxlc3MgdGhhbiA0MCUgb2YgdGhlIHRocm91Z2hw dXQgd2Ugc2F3IHVuZGVyIC1TUDEuIFdpdGggdGhhdCBsb3cKcGVyZm9ybWFuY2UsIHdlIGhh ZCBhIDQgZGF5IHJ1biB3aXRob3V0IGFueSBkcm9wcGVkIFJQQyByZXF1ZXN0LiBCdXQgd2Ug ZG9uJ3QKa25vdyB0aGUgZXJyb3IgcmF0ZSB3aXRob3V0IHRoZSBwYXRjaGVzIHVuZGVyIHRo ZXNlIGNvbmRpdGlvbnMuIFNvIHdlIGNhbid0CmdpdmUgYW4gby5rLiBmb3IgdGhlIHBhdGNo ZXMgeWV0LgoKQ3VycmVudGx5IHdlIHRyeSB0byBmaW5kIHRoZSByZWFzb24gZm9yIHRoZSBk aWZmZXJlbnQgYmVoYXZpb3Igb2YgU1AxIGFuZCBTUDIKCkJvZG8K
Hi. Bodo.
We are facing issues with respect to the SUNRPC cache. In our case we have two targets connected back-to-back NFS Server: Kernel version, 2.6.35
At times, when Client tries to connect to the Server it stucks for very long duration and keeps on trying to mount.
When we try to figure out using logs, we checked that client was not getting response of FSINFO request.
Further, by debugging we found that the request was getting dropped at the SERVER, so this request was not being served.
In the code we reached this point:
svcauth_unix_set_client()->
gi = unix_gid_find(cred->cr_uid, rqstp);
switch (PTR_ERR(gi)) {
case -EAGAIN:
return SVC_DROP;This path is related with the SUNRPC cache management.
When we remove this UNIX_GID_FIND path from our code, there is no problem.
When we try to figure the possible related problems as per our scneario, We found that you have faced similar issue for RACE in the cache.
Can you please suggest what could be the problem so that we can check further ?
Or from the solution if you encounter the similar situation. can you please suggest on the possible patches for 2.6.35 - which we can try in our environment ?
We will be highly grateful.
Thanks
2013/4/20, Bodo Stroesser <[email protected]>:
Quoted 65 lines · On 05 Apr 2013 23:09:00 +0100 J. Bruce Fields <[email protected]> wrote:
Quoted 34 lines · Bodo Stroesser
Quoted 33 lines
Quoted 31 lines · On 05 Apr 2013 14:40:00 +0100 J. Bruce Fields <[email protected]> wrote:
Quoted 16 lines · Bodo Stroesser
There is no reason for apologies. The thread meanwhile seems to be a bit
confusing :-)Current state is:
- Neil Brown has created two series of patches. One for SLES11-SP1 and a
second one for -SP2- AFAICS, the series for -SP2 will match with mainline also.
- Today I found and fixed the (hopefully) last problem in the -SP1 series.
My test using this patchset will run until Monday.Quoted 5 lines
- Provided the test on SP1 succeeds, probably on Tuesday I'll start to test
the patches for SP2 (and mainline). If it runs fine, we'll have a
tested
patchset not later than Mon 15th.OK, great, as long as it hasn't just been forgotten!
I'd also be curious to understand why we aren't getting a lot of complaints about this from elsewhere.... Is there something unique about your setup? Do the bugs that remain upstream take a long time to
reproduce?Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
--b.
It's no secret, what we are doing. So let me try to explain:
Thanks for the detailed explanation! I'll look forward to the patches.
--b.
Let me give an intermediate result:
The test of the -SP1 patch series succeeded.
We started the test of the -SP2 (and mainline) series on Tue, 9th, but had no
success.
We did _not_ find a problem with the patches, but under -SP2 our test scenario
has less than 40% of the throughput we saw under -SP1. With that low performance, we had a 4 day run without any dropped RPC request. But we don't
know the error rate without the patches under these conditions. So we can't give an o.k. for the patches yet.Currently we try to find the reason for the different behavior of SP1 and SP2
Bodo
Hi.
Sorry for interrupt.
I fixed my issue using this patch(nfsd4: fix hang on fast-booting nfs servers). it was different issue with this subject on current mail.
Thanks.
2013/5/10, Namjae Jeon <[email protected]>:
Quoted 114 lines
Hi. Bodo.
We are facing issues with respect to the SUNRPC cache. In our case we have two targets connected back-to-back NFS Server: Kernel version, 2.6.35
At times, when Client tries to connect to the Server it stucks for very long duration and keeps on trying to mount.
When we try to figure out using logs, we checked that client was not getting response of FSINFO request.
Further, by debugging we found that the request was getting dropped at the SERVER, so this request was not being served.
In the code we reached this point:
svcauth_unix_set_client()->gi = unix_gid_find(cred->cr_uid, rqstp); switch (PTR_ERR(gi)) { case -EAGAIN: return SVC_DROP;This path is related with the SUNRPC cache management.
When we remove this
UNIX_GID_FINDpath from our code, there is no problem.When we try to figure the possible related problems as per our scneario, We found that you have faced similar issue for RACE in the cache.
Can you please suggest what could be the problem so that we can check further ?Or from the solution if you encounter the similar situation. can you please suggest on the possible patches for 2.6.35 - which we can try in our environment ?
We will be highly grateful.
Thanks
2013/4/20, Bodo Stroesser <[email protected]>:
Quoted 71 lines
Quoted 37 lines · On 05 Apr 2013 23:09:00 +0100 J. Bruce Fields <[email protected]> wrote:
Quoted 36 lines · Bodo Stroesser
Quoted 34 lines · On 05 Apr 2013 14:40:00 +0100 J. Bruce Fields <[email protected]> wrote:
Quoted 17 lines · Bodo Stroesser
There is no reason for apologies. The thread meanwhile seems to be a
bit
confusing :-)Current state is:
- Neil Brown has created two series of patches. One for SLES11-SP1 and a
second one for -SP2- AFAICS, the series for -SP2 will match with mainline also.
- Today I found and fixed the (hopefully) last problem in the -SP1 series.
My test using this patchset will run until Monday.Quoted 7 lines
- Provided the test on SP1 succeeds, probably on Tuesday I'll start
to test
the patches for SP2 (and mainline). If it runs fine, we'll have
a
tested
patchset not later than Mon 15th.OK, great, as long as it hasn't just been forgotten!
I'd also be curious to understand why we aren't getting a lot of complaints about this from elsewhere.... Is there something unique about your setup? Do the bugs that remain upstream take a long time to
reproduce?Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
--b.
It's no secret, what we are doing. So let me try to explain:
Thanks for the detailed explanation! I'll look forward to the patches.
--b.
Let me give an intermediate result:
The test of the -SP1 patch series succeeded.
We started the test of the -SP2 (and mainline) series on Tue, 9th, but had
no
success.
We did _not_ find a problem with the patches, but under -SP2 our test scenario
has less than 40% of the throughput we saw under -SP1. With that low performance, we had a 4 day run without any dropped RPC request. But we don't
know the error rate without the patches under these conditions. So we can't
give an o.k. for the patches yet.Currently we try to find the reason for the different behavior of SP1 and SP2
Bodo