16-CPU #s for lockfree rtcache (rt_rcu)

As promised, here is the kernprof data from the simulated test suggested by Dave. The test uses 32 processes and dest addresses for rtcache lookup are random, changing after every 5 packets. The measurements were done in a 16 CPU NUMA-Q.

2.5.3 : ip_route_output_key [c01bab8c]: 12166 2.5.3+rt_rcu : ip_route_output_key [c01bb084]: 6027

I have seen moderately significant profile counts
for ip_route_input() in preliminary webserver benchmark runs. It is not however clear to me that bucket lock cache line bouncing is the reason behind it. That one needs more investigation.

Thanks

Dipankar Sarma  <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.

From: Dipankar Sarma <[email protected]>
Date: Fri, 17 May 2002 19:21:16 +0530

2.5.3 : ip_route_output_key [c01bab8c]: 12166 2.5.3+rt_rcu : ip_route_output_key [c01bb084]: 6027

Thanks for doing the testing. Are you able to do this test on some 4 or 8 processor non-NUMA system?

Basically halfing the profile hits for this function
is wonderful and I'd love to see how much of this translates to a non-NUMA system.

I have seen moderately significant profile counts
for ip_route_input() in preliminary webserver benchmark runs. It is not however clear to me that bucket lock cache line bouncing is the reason behind it. That one needs more investigation.

This is where most of the routing heavy work is done on
a web server, so this doesn't surprise me. Once packet
is input and routed, we have destination and just grab a reference to and use it for output back to that remote host.

Yes, but may not have been the same test. We have been doing various configurations for this test. One fallout of using large number of dest addresses is that we have frequent neighbor table garbage collection which results in a lot of lock contentions. By slowing down the packet rate and/or increasing the gc threshold, we can avoid this. How realistic is this ? If we avoid frequent
gc, we see better gains in route lookup. With frequent gc, the speedup was of about 22% for an 8 cpu SMP, IIRC. I will rerun the tests tomorrow or monday to get both sets of numbers for 8-cpu SMP.

Thanks

Dipankar Sarma  <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.

From: Dipankar Sarma <[email protected]>
Date: Fri, 17 May 2002 21:44:33 +0530

I will rerun the tests tomorrow or monday to get both sets of numbers for 8-cpu SMP.

Provide the data, it will be interesting.

Yes, that is a another problem needs addressing.

BTW, do_gettimeofday() also shows up moderately significant in profile of 8-CPU webserver benchmark. I will address xtime_lock separately.

Thanks

Dipankar Sarma  <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.

I bet the numbers would be much better if the x86 do_gettimeofday() was converted to a lockless version first ... Currently it is bouncing around its readlock for every incoming packet.

-Andi

From: Andi Kleen <[email protected]>
Date: 17 May 2002 21:25:16 +0200

I bet the numbers would be much better if the x86 do_gettimeofday() was converted to a lockless version first ... Currently it is bouncing around its readlock for every incoming packet.

That is true. But right now we are trying to analyze the effects of his patch all by itself.