From: Al Boldi Subject: Re: Massive slowdown when re-querying large nfs dir Date: Wed, 7 Nov 2007 12:36:26 +0300 Message-ID: <200711071236.26780.a1426z@gawab.com> References: <200711050758.38090.a1426z@gawab.com> <20071106221939.cfa79f9e.akpm@linux-foundation.org> <18225.26935.146395.366451@notabene.brown> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org To: Neil Brown , Andrew Morton Return-path: Received: from [212.12.190.111] ([212.12.190.111]:56399 "EHLO raad.intranet" rhost-flags-FAIL-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1750695AbXKGJgx (ORCPT ); Wed, 7 Nov 2007 04:36:53 -0500 In-Reply-To: <18225.26935.146395.366451@notabene.brown> Content-Disposition: inline Sender: linux-fsdevel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org Neil Brown wrote: > On Tuesday November 6, akpm@linux-foundation.org wrote: > > > On Tue, 6 Nov 2007 14:28:11 +0300 Al Boldi wrote: > > > Al Boldi wrote: > > > > There is a massive (3-18x) slowdown when re-querying a large nfs dir > > > > (2k+ entries) using a simple ls -l. > > > > > > > > On 2.6.23 client and server running userland rpc.nfs.V2: > > > > first try: time -p ls -l <2k+ entry dir> in ~2.5sec > > > > more tries: time -p ls -l <2k+ entry dir> in ~8sec > > > > > > > > first try: time -p ls -l <5k+ entry dir> in ~9sec > > > > more tries: time -p ls -l <5k+ entry dir> in ~180sec > > > > > > > > On 2.6.23 client and 2.4.31 server running userland rpc.nfs.V2: > > > > first try: time -p ls -l <2k+ entry dir> in ~2.5sec > > > > more tries: time -p ls -l <2k+ entry dir> in ~7sec > > > > > > > > first try: time -p ls -l <5k+ entry dir> in ~8sec > > > > more tries: time -p ls -l <5k+ entry dir> in ~43sec > > > > > > > > Remounting the nfs-dir on the client resets the problem. > > > > > > > > Any ideas? > > > > > > Ok, I played some more with this, and it turns out that nfsV3 is a lot > > > faster. But, this does not explain why the 2.4.31 kernel is still > > > over 4-times faster than 2.6.23. > > > > > > Can anybody explain what's going on? > > > > Sure, Neil can! ;) Thanks Andrew! > Nuh. > He said "userland rpc.nfs.Vx". I only do "kernel-land NFS". In these > days of high specialisation, each line of code is owned by a different > person, and finding the right person is hard.... > > I would suggest getting a 'tcpdump -s0' trace and seeing (with > wireshark) what is different between the various cases. Thanks Neil for looking into this. Your suggestion has already been answered in a previous post, where the difference has been attributed to "ls -l" inducing lookup for the first try, which is fast, and getattr for later tries, which is super-slow. Now it's easy to blame the userland rpc.nfs.V2 server for this, but what's not clear is how come 2.4.31 handles getattr faster than 2.6.23? Thanks! -- Al