FYI: readdirplus responsiveness problem after 2.4 -> 2.6 client upgrade

DISCUSSION Frank van Maarseveen 2 participants 3 messages Jul 11, 2005 – Jul 12, 2005

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Frank van Maarseveen

author

2005-07-11 16:51 UTC #0 lore raw

Server is tru64 (OSF1 server V5.1 1885 alpha) using advfs.

client:
After replacing a 2.4.29 kernel by a 2.6 (.11) we started to notice unresponsiveness in bash command- and name completion once in a while. This still persists in 2.6.12.2.

A few quick investigations pointed to the readdir -> readdirplus change. The following tcpdump snippets illustrates this (first column is differential time in usec with previous packet):

example 1:
------
000093 ip 230: client.3377456114 > server.nfs: 188 readdirplus fh 4079,976512/13921 512 bytes @ 0x000000000 (DF) 645029 ip 1514: server.nfs > client.3377456114: reply ok 1472 readdirplus (frag 19592:1480@0+) 000058 ip 778: server > client: udp (frag 19592:744@1480)
------

example 2:
------
000094 ip 230: client.743433202 > server.nfs: 188 readdirplus fh 4079,976512/404 512 bytes @ 0x000000000 (DF) 102655 ip 230: client.743433202 > server.nfs: 188 readdirplus fh 4079,976512/404 512 bytes @ 0x000000000 (DF) 205974 ip 230: client.743433202 > server.nfs: 188 readdirplus fh 4079,976512/404 512 bytes @ 0x000000000 (DF) 248905 ip 1514: server.nfs > client.743433202: reply ok 1472 readdirplus (frag 58739:1480@0+) 000085 ip 1106: server > client: udp (frag 58739:1072@1480)

000128 ip 1514: server.nfs > client.743433202: reply ok 1472 readdirplus (frag 58740:1480@0+) 000057 ip 230: client.760210418 > server.nfs: 188 readdirplus fh 4079,976512/404 512 bytes @ 0x0000001e4 (DF) 000028 ip 1106: server > client: udp (frag 58740:1072@1480)

000125 ip 1514: server.nfs > client.743433202: reply ok 1472 readdirplus (frag 58741:1480@0+) 000088 ip 1106: server > client: udp (frag 58741:1072@1480)

022016 ip 506: server.nfs > client.760210418: reply ok 464 readdirplus
------

AFAICS this only shows a (sometimes) slow responding server. In the second example the client appears to be retransmitting much sooner than in the first example. Don't know if this can be true or that the differential timestamps are fooling me.

In practice there are a significant number of cases without retransmission when the server appears unresponsive to the user with readdirplus. Since it is induced by bash name completion it is very visible to the user:
occasionally command-completion takes 3..10 seconds now to give an idea. Name completion for arguments is hurt too sometimes.

A quick peek into the source suggested:

- reduce NFS_LIMIT_READDIRPLUS?
- "recognize" name-completion kind of use and revert
NFS_INO_ADVISE_RDPLUS?

Other options (except replacing the server)?

Signature · 11 lines

Frank


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP, 
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Trond Myklebust

2005-07-11 17:17 UTC #1 lore raw

m=C3=A5 den 11.07.2005 Klokka 18:51 (+0200) skreiv Frank van Maarseveen:

Quoted 1 line

In practice there are a significant number of cases without retransmissio=

Quoted 9 lines

when the server appears unresponsive to the user with readdirplus. Since it is induced by bash name completion it is very visible to the user:
occasionally command-completion takes 3..10 seconds now to give an idea. Name completion for arguments is hurt too sometimes.
=20
A quick peek into the source suggested:
=20
(quoted diff) +0 -2
-	reduce NFS_LIMIT_READDIRPLUS?
-	"recognize" name-completion kind of use and revert

Quoted 3 lines

NFS_INO_ADVISE_RDPLUS?
=20
Other options (except replacing the server)?

I suppose you could mask out NFS_CAP_READDIRPLUS on the client, but it sounds very strange that the OSF implementation should be so incredibly slow. Is there nothing in the administrators manual about tuning this sort of operation?

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual core and dual graphics technology at this free one hour event hosted by HP, AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Frank van Maarseveen

author

2005-07-12 15:56 UTC #2 lore raw

Quoted 2 lines · Trond Myklebust

m=E5 den 11.07.2005 Klokka 18:51 (+0200) skreiv Frank van Maarseveen:
Quoted 1 line
In practice there are a significant number of cases without retransmiss=

ion

Quoted 9 lines

Quoted 4 lines
when the server appears unresponsive to the user with readdirplus. Since it is induced by bash name completion it is very visible to the user:
occasionally command-completion takes 3..10 seconds now to give an idea. Name completion for arguments is hurt too sometimes.
=20
I suppose you could mask out NFS_CAP_READDIRPLUS on the client, but it sounds very strange that the OSF implementation should be so incredibly slow. Is there nothing in the administrators manual about tuning this sort of operation?

Maybe. I'm not an tru64 expert. But I do know that the server in question is underpowered, e.g. lookups, getattr are a bit slow at times as well. Anyway, the "user experience" seems better with readdirplus disabled on the client so I decided to do that.

--=20
Frank