Skip to contentSkip to conversation

knfsd: cache ipmap per TCP socket

PATCH New Greg Banks 1 participant 1 message Aug 8, 2006

Subsystems (3)
lore
Share & tools
lore

Greg Banks

author
#0 lore raw

PATCH 1/1 knfsd: cache ipmap per TCP socket

Oops, looks like I forgot to cc the list on this one. --

knfsd: speed up high call-rate workloads by caching the struct ip_map for the peer on the connected struct svc_sock instead of looking it up in the ip_map cache hashtable on every call. This helps workloads using AUTH_SYS authentication over TCP.

Testing was on a 4 CPU 4 NIC Altix using 4 IRIX clients, each with 16 synthetic client threads simulating an rsync (i.e. recursive directory listing) workload reading from an i386 RH9 install image (161480 regular files in 10841 directories) on the server. That tree is small enough to fill in the server's RAM so no disk traffic was involved. This setup gives a sustained call rate in excess of 60000 calls/sec before being CPU-bound on the server.

Profiling showed strcmp(), called from ip_map_match(), was taking 4.8% of each CPU, and ip_map_lookup() was taking 2.9%. This patch drops both contribution into the profile noise.

Note that the above result overstates this value of this patch for most workloads. The synthetic clients are all using separate IP addresses, so there are 64 entries in the ip_map cache hash. Because the kernel measured contained the bug fixed in commit

commit 1f1e030bf75774b6a283518e1534d598e14147d4

and was running on 64bit little-endian machine, probably all of those 64 entries were on a single chain, thus increasing the cost of ip_map_lookup().

With a modern kernel you would need more clients to see the same amount of performance improvement. This patch has helped to scale knfsd to handle a deployment with 2000 NFS clients.

---

 include/linux/sunrpc/cache.h   |    5 +++
 include/linux/sunrpc/svcauth.h |    1 
 include/linux/sunrpc/svcsock.h |    3 +
 net/sunrpc/svcauth_unix.c      |   47 ++++++++++++++++++++++++++++--
 net/sunrpc/svcsock.c           |    2 +
 5 files changed, 55 insertions(+), 3 deletions(-)
include/linux/sunrpc/svcsock.h +3 -0
Index: linux-2.6.18-rc2/include/linux/sunrpc/svcsock.h
===================================================================
@@ -54,6 +54,9 @@ struct svc_sock {
 	int			sk_reclen;	/* length of record */
 	int			sk_tcplen;	/* current read length */
 	time_t			sk_lastrecv;	/* time of last received request */
+
+	/* cache of various info for TCP sockets */
+	void			*sk_info_authunix;
 };
 
 /*
net/sunrpc/svcauth_unix.c +44 -3

71 lines not on this page · raw message

net/sunrpc/svcsock.c +2 -0
Index: linux-2.6.18-rc2/net/sunrpc/svcsock.c
===================================================================
@@ -1635,6 +1635,8 @@ svc_delete_socket(struct svc_sock *svsk)
 			sockfd_put(svsk->sk_sock);
 		else
 			sock_release(svsk->sk_sock);
+		if (svsk->sk_info_authunix != NULL)
+			svcauth_unix_info_release(svsk->sk_info_authunix);
 		kfree(svsk);
 	} else {
 		spin_unlock_bh(&serv->sv_lock);
include/linux/sunrpc/svcauth.h +1 -0
Index: linux-2.6.18-rc2/include/linux/sunrpc/svcauth.h
===================================================================
@@ -126,6 +126,7 @@ extern struct auth_domain *auth_domain_f
 extern struct auth_domain *auth_unix_lookup(struct in_addr addr);
 extern int auth_unix_forget_old(struct auth_domain *dom);
 extern void svcauth_unix_purge(void);
+extern void svcauth_unix_info_release(void *);
 
 static inline unsigned long hash_str(char *name, int bits)
 {
include/linux/sunrpc/cache.h +5 -0
Index: linux-2.6.18-rc2/include/linux/sunrpc/cache.h
===================================================================
@@ -163,6 +163,11 @@ static inline void cache_put(struct cach
 	kref_put(&h->ref, cd->cache_put);
 }
 
+static inline int cache_valid(struct cache_head *h)
+{
+	return (h->expiry_time != 0 && test_bit(CACHE_VALID, &h->flags));
+}
+
 extern int cache_check(struct cache_detail *detail,
 		       struct cache_head *h, struct cache_req *rqstp);
 extern void cache_flush(void);

Greg.

Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs