Locking comment on shrink_caches()
Quoted 7 lines · In article <[email protected]> David S. Miller wrote:
From: Rik van Riel <[email protected]>
Date: Tue, 25 Sep 2001 17:24:21 -0300 (BRST)Or were you measuring loads which are mostly read-only ?
When Kanoj Sarcar was back at SGI testing 32 processor Origin MIPS systems,
pagecache_lockwas at the top.
John Hawkes from SGI had published some AIM7 numbers that showed pagecache_lock to be a bottleneck above 4 processors. At 32 processors, half the CPU cycles were spent on waiting for pagecache_lock. The thread is at -
http://marc.theaimsgroup.com/?l=lse-tech&m=98459051027582&w=2
Thanks
Dipankar
Signature · 2 lines
Dipankar Sarma <[email protected]> Project: http://lse.sourceforge.net Linux Technology Center, IBM Software Lab, Bangalore, India.
Quoted 10 lines · Dipankar Sarma
Quoted 7 lines · In article <[email protected]> David S. Miller wrote:
From: Rik van Riel <[email protected]>
Date: Tue, 25 Sep 2001 17:24:21 -0300 (BRST)Or were you measuring loads which are mostly read-only ?
When Kanoj Sarcar was back at SGI testing 32 processor Origin MIPS systems,
pagecache_lockwas at the top.
Quoted 7 lines
John Hawkes from SGI had published some AIM7 numbers that showed
pagecache_lockto be a bottleneck above 4 processors. At 32 processors, half the CPU cycles were spent on waiting forpagecache_lock. The thread is at -http://marc.theaimsgroup.com/?l=lse-tech&m=98459051027582&w=2
That's NUMA hardware. The per-hashqueue locking change made
a big improvement on that hardware. But when it was used on Intel hardware it made no measurable difference at all.
Sorry, but the patch adds compexity and unless a significant throughput benefit can be demonstrated on less exotic hardware, why use it?
Here are kumon's test results from March, with and without the hashed lock patch:
-------- Original Message --------
Subject: Re: [Fwd: Re: [Lse-tech] AIM7 scaling, pagecache_lock, multiqueue scheduler]
Date: Thu, 15 Mar 2001 18:03:55 +0900
From: [email protected]
Reply-To: [email protected]
To: Andrew Morton <[email protected]>
CC: [email protected], [email protected],John Hawkes <[email protected]>,[email protected]
In-Reply-To: <[email protected]>,<[email protected]><[email protected]><[email protected]>
OK, the followings are a result of our brief measurement with WebBench (mindcraft type) of 2.4.2 and 2.4.2+pcl .
Workload: WebBench 3.0 (static get)
Machine: Profusion 8way 550MHz/1MB cache 1GB mem.
Server: Apache 1.3.9-8 (w/ SINGLE_LISTEN_UNSERIALIZED_ACCEPT)
obtained from RedHat.
Clients: 32 clients each has 2 requesting threads.
The following number is Request per sec.
242 242+pcl ratio
-------------------------------------
1SMP 1,603 1,584 0.99
2(1+1)SMP 2,443 2,437 1.00
4(1+3)SMP 4,420 4,426 1.00
8(4+4)SMP 5,381 5,400 1.00
#No idle time observed in the 1 to 4 SMP runs.
#Only 8 SMP cases shows cpu-idle time, but it is about 2.1-2.8% of the #total CPU time.
Note: The load of two buses of Profusion system isn't balance, because the number of CPUs on each bus is unbalance.
Summary:
From the above brief test, (+pcl) patch doens't show the measurable
performance gain.
-
From: Andrew Morton <[email protected]>
Date: Tue, 25 Sep 2001 22:31:32 -0700
Here are kumon's test results from March, with and without the hashed lock patch:
Please elaborate on what the webbench-3.0 static gets was really doing.
Was this test composed of multiple accesses to the same or a small set of files? If so, that is indeed the case where the page cache locking patches won't help at all.
The more diversified the set of files being accessed, the greater the gain from the locking changes. You have to encourage the cpus at least have a chance at accessing different hash chains :-)
Franks a lot,
David S. Miller
[email protected]
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Dipankar Sarma
John Hawkes from SGI had published some AIM7 numbers that showed
pagecache_lockto be a bottleneck above 4 processors. At 32 processors, half the CPU cycles were spent on waiting forpagecache_lock. The thread is at -http://marc.theaimsgroup.com/?l=lse-tech&m=98459051027582&w=2
Quoted 7 lines
That's NUMA hardware. The per-hashqueue locking change made
a big improvement on that hardware. But when it was used on Intel hardware it made no measurable difference at all.Sorry, but the patch adds compexity and unless a significant throughput benefit can be demonstrated on less exotic hardware, why use it?
I agree that on NUMA systems, contention and lock wait times degenerate non-linearly thereby skewing the actual impact.
IIRC, there were discussions on lse-tech about pagecache_lock and dbench numbers published by Juergen Doelle (on 8way Intel) and Anton Blanchard on 16way PPC. Perhaps they can shed some light on this.
Thanks
Dipankar
Signature · 2 lines
Dipankar Sarma <[email protected]> Project: http://lse.sourceforge.net Linux Technology Center, IBM Software Lab, Bangalore, India.
From: "Andrew Morton" <[email protected]>
Quoted 2 lines
Quoted 2 lines
John Hawkes from SGI had published some AIM7 numbers that showed
pagecache_lockto be a bottleneck above 4 processors. At 32
processors,
Quoted 9 lines
Quoted 5 lines
half the CPU cycles were spent on waiting for
pagecache_lock. The thread is at -http://marc.theaimsgroup.com/?l=lse-tech&m=98459051027582&w=2
That's NUMA hardware. The per-hashqueue locking change made
a big improvement on that hardware. But when it was used on Intel hardware it made no measurable difference at all.
More specifically, that was on SGI Origin2000 32p mips64 ccNUMA hardware. The pagecache_lock bottleneck is substantially less on SGI Itanium ccNUMA hardware running those AIM7 workloads. I'm seeing moderately significant contention on the Big Kernel Lock, mostly from sys_lseek() and ext2_get_block().
John Hawkes
[email protected]