smp cputime issues
Quoted 26 lines
smp cputime issues
hello,
we are encountering some weird timing behaviour on our linux cluster.
specifically: when running 2 copies of selected programs on a dual-cpu system, the cputime reported for each process is up to 25% higher than when the processes are run on their own. however, if running two different jobs on the same machine, both complete with a cputime equal to when run individually. sample timing output attached.
profiling confirms that everything slows down approximately to scale. the results reproduce on a range of different machines (see below).
additional specifications:
- kernel version 2.4.16 (with apic enabled)
- chipsets: apollo pro 133, apollo pro 266,
intel i860, serverworks LE
- all jobs requires less than 1/10 of physical memory
- no significant disk i/o takes place
- timing withdtime(),/usr/bin/timeand shell built-in time
- this behavior is NOT seen for all applications. the worst "offender" spends most of its time doing linear algebra.
Quoted 2 lines
ideas or info-pointers appreciated. more specs available on request.
two points. First for clarification - do you see the effects also on
elapsed time? Or do you say that the CPU time reporting is screwed?
Second - you mention that you see the effect mainly on linear algebra
stuff. Could it be that you are memory bandwidth limited if you run two of them together? Are you using Intel CPUs (my guess) which have the FSB concept that may make memory bandwidth scaling a problem, or AMD Athlons which use the Alpha/EV6 bus and should be a bit more friendly.
Finally, how big is "1/10th of physical" memory? What kind of memory.
Martin
Signature · 7 lines
+-----------------------------------------------------+ |Martin Knoblauch | |-----------------------------------------------------| |http://www.knobisoft.de/cats | |-----------------------------------------------------| |e-mail: [email protected] | +-----------------------------------------------------+
Quoted 5 lines
Second - you mention that you see the effect mainly on linear algebra
stuff. Could it be that you are memory bandwidth limited if you run two of them together? Are you using Intel CPUs (my guess) which have the FSB concept that may make memory bandwidth scaling a problem, or AMD Athlons which use the Alpha/EV6 bus and should be a bit more friendly.
Hmmm ... linear algebra ... are you by any chance using Atlas? Atlas is highly optimized for the chips and as many other architectural features as it can discover, such as cache size. I'm sure a well-tuned Atlas application is quite capable of bending a machine to its own purposes, quite possibly to the discomfort of other users attempting to use the system. If the issue is sharing of resources between the linear algebra code and other users, perhaps the thing to do is get Atlas, if you're not currently using it, and then "nice" the linear algebra code.
I run Atlas on my (UP) 1.333 GHz Athlon Thunderbird and it screams. I can get 4+ GFLOPS in the 3DNOW 32-bit code and well over 1 GFLOP in 64 bits. --
M. Edward Borasky