Skip to contentSkip to conversation

2.4.20-rc1 NFS/TCP client (still) hangs running dbench 2.0

PATCH New Andrew Ryan 3 participants 5 messages Nov 1, 2002 – Nov 3, 2002 1 message not in the archive

Subsystems (3)
Networking NFS client NFS server
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andrew Ryan

author
#1 lore raw
Quoted 7 lines · you

" " == Andrew Ryan <[email protected]> writes:
> 2.4.20rc1 still exhibits the same identical problem as > 2.4.19+kmap1 patch which I identified a few weeks ago. I guess > that patch was applied to 2.4.20, since it doesn't appear on > Trond's patch page for 2.4.20.

Does the following patch work for you?

So far so good on the crashes. I'm able to get through a complete run of dbench using TCP mounts on 2.4.20rc1, which I haven't been able to do before this.

However, according to nfsstat -c's "Client rpc stats", I'm getting around a 10:1 ratio of retransmits to calls. That's bad. I get 0 retransmits when using 2.4.19+kmap1, and any previous kernel I've ever tried TCP mounting with. I checked 2.4.20rc1-vanilla and it does the same thing, so it's not the latest patch.

andrewr

-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Andrew Ryan <[email protected]> writes:

> However, according to nfsstat -c's "Client rpc stats", I'm > getting around a 10:1 ratio of retransmits to calls. That's > bad. I get 0 retransmits when using 2.4.19+kmap1, and any > previous kernel I've ever tried TCP mounting with. I checked > 2.4.20rc1-vanilla and it does the same thing, so it's not the > latest patch.

It is probably just the (known) accounting error in the 'retransmit' statistics. Try applying the patch

http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif

if it bothers you.

Note: the TCP timeout + retransmits will (unlike UDP) be of the order of 60 seconds, so if these were real retransmits, you'd be seeing a *very* noticeable slowdown...

Cheers,
Trond

-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#3 lore raw
Quoted 7 lines · Trond Myklebust

It is probably just the (known) accounting error in the 'retransmit' statistics. Try applying the patch

http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif

if it bothers you.

Yes, you are right, it seems the counter is broken, and performance is fine. I did bonnie++ runs with 2.4.20rc1 kernels with and without the call_start patch, and didn't see any significant difference.

But, it does bother me since I want to be able to diagnose network errors if I ever have them, and with a busted counter it will be impossible to tell what my real error rate is. Can the call_start patch go to Marcelo for 2.4.20rc2?

After a bonnie run with the call_start patch, I get this, which is much better :)

$ nfsstat -c
Client rpc stats:
calls      retrans    authrefrsh
1648050    0          0

thanks,
andrew

-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Andrew Ryan writes:

Quoted 6 lines · Trond Myklebust

It is probably just the (known) accounting error in the 'retransmit' statistics. Try applying the patch

http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif

if it bothers you.

Quoted 8 lines

Yes, you are right, it seems the counter is broken, and performance is fine. I did bonnie++ runs with 2.4.20rc1 kernels with and without the call_start patch, and didn't see any significant difference.

But, it does bother me since I want to be able to diagnose network errors if I ever have them, and with a busted counter it will be impossible to tell what my real error rate is. Can the call_start patch go to Marcelo for 2.4.20rc2?

I'm an end user with approximately 3000 Linux NFS clients. I will really appreciate correct counters when I'm debugging nfs server, client and network problems. I've had to ignore spurious counts of nfs retrans while debugging network issues.

I hope this patch will be in 2.4.20rc2. Thanks to all of you for providing an excellent client-side implementation. And I am serious.

Quoted 7 lines

After a bonnie run with the call_start patch, I get this, which is much better :)

$ nfsstat -c
Client rpc stats:
calls      retrans    authrefrsh
1648050    0          0
Signature · 14 lines
Quentin Fennessy   		[email protected]	
				Office: 512.602.3873
				Cell:   512.694.7489




-------------------------------------------------------
This SF.net email is sponsored by: ApacheCon, November 18-21 in
Las Vegas (supported by COMDEX), the only Apache event to be
fully supported by the ASF. http://www.apachecon.com
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#5 lore raw

PATCH 1/1 another kmap imbalance in 2.4.x/2.5.x RPC

In reply to Andrew Ryan
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Andrew Ryan <[email protected]> writes:

> So far so good on the crashes. I'm able to get through a > complete run of dbench using TCP mounts on 2.4.20rc1, which I > haven't been able to do before this.

Marcelo, Linus

We've uncovered yet another kmap imbalance in the new RPC code. This
looks like it might be the last one (my debugging printks have been unable to unearth any more). One line fix + 4 line comment appended. Please apply to both 2.4.20-rc1 and 2.5.45...

Cheers,
Trond

net/sunrpc/xdr.c +5 -0
diff -u --recursive --new-file linux-2.4.20-rc1/net/sunrpc/xdr.c linux-2.4.20-kmap_fix/net/sunrpc/xdr.c
@@ -244,6 +244,11 @@
 		pglen -= base;
 		base  += xdr->page_base;
 		ppage += base >> PAGE_CACHE_SHIFT;
+		/* Note: The offset means that the length of the first
+		 * page is really (PAGE_CACHE_SIZE - (base & ~PAGE_CACHE_MASK)).
+		 * In order to avoid an extra test inside the loop,
+		 * we bump pglen here, and just subtract PAGE_CACHE_SIZE... */
+		pglen += base & ~PAGE_CACHE_MASK;
 	}
 	for (;;) {
 		flush_dcache_page(*ppage);

-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs