Skip to contentSkip to conversation

Multiprocessor Control Interfaces

RFC Jason Baietto 2 participants 3 messages Dec 10, 2001 – Dec 11, 2001

lore
Share & tools
lore
#0 lore raw

Hello All,

I'm currently working on adding multiprocessor control interfaces to Linux. My current efforts can be found here:

http://www.ccur.com/realtime/oss

These are clean-room implementations of similar tools that have been available in our proprietary *nix for quite some time, and so the interfaces have a fair amount of mileage under their belts. Note that the scope is somewhat wider than just MP.

There has been some discussion of "chaff" and other interfaces recently on this list, so in an effort to hopefully move towards a standard more quickly I've gotten permission from my employer to GPL the code I've written. I'm very interested in comments and feedback on any or all of this work.

Here's the README file from the package:

This package contains:

run(1)
A multiprocessor control command line tool.

mpadvise(3)
A multiprocessor control library interface.

These services rely upon Robert Love's CPU Affinity patch (version 2.4.16-1 was used for testing) which is available here:

http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/

To build the code, simply unpack it and type "make". The code has been tested on Red Hat 7.1 and 7.2 systems, though it is still fairly new and almost certainly contains bugs.

An attempt was made to abstract the "cpuset" representation of the current system in order to have binaries that in theory could work on systems with more than 32 cpus. For this to work, the run(1) command would need to be linked against a shared mpadvise(3) library (currently only a static library is made).

This code is being released in the hopes that it will become the basis for the Linux multiprocessor control standard interfaces. I am very interested in getting feedback on this package, so please contact me via email or LKML if you have any.

This source code is licensed under the GNU GPL Version 2. Copyright (C) 2001 Concurrent Computer Corporation

--
Jason Baietto
[email protected]
http://www.ccur.com/realtime/oss

#1 lore raw
Quoted 9 lines · Jason Baietto

I'm currently working on adding multiprocessor control interfaces to Linux. My current efforts can be found here:

http://www.ccur.com/realtime/oss

These are clean-room implementations of similar tools that have been available in our proprietary *nix for quite some time, and so the interfaces have a fair amount of mileage under their belts. Note that the scope is somewhat wider than just MP.

Ahh, very neat. This is a useful tool.

One idea would be to allow users to set CPUs processes _can't_ run on. On high-end systems sometimes a CPU is affined to a particular IRQ (say, network interface). Another situation is where you bind a RT task to a given CPU. In these situations, you want everything else to _not_ run on the CPUs. I.e., `run --bind=!1' (note its easy to generate the bitmask here too, by ANDing the inverse of the given CPU against -1).

At any rate, what is needed most is to standardize on an interface for these scheduling mechanisms. I guess its just CPU affinity we have to go ... since not much progress was made of my (proc-based) method vs. Ingo's (syscall-based) method, at this point either of the two being merged would make me happy.

Quoted 4 lines

These services rely upon Robert Love's CPU Affinity patch (version 2.4.16-1 was used for testing) which is available here:

http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/

I assume you have no problems with it ... I think I'd like to add the change that the CPUs reported correspond to the physical CPU number and not the logical value we derive. On x86 this won't make a difference, but its a proper method I suspect.

Robert Love

#2 lore raw
Quoted 3 lines · Robert Love

One idea would be to allow users to set CPUs processes _can't_ run on.

A technique that many of our customers currently use with the run command is to shield a sensitive CPU in rc.sysinit by simply using run to bias the init process. For example, on a four CPU system, the following command:

   run --bias 1-3 --pid 1

would bias init to only run on CPUs 1, 2 or 3. Since all children inherit CPU affinity, this effectively makes CPU 0 off limits for any process that doesn't explicitly bias itself to CPU 0 via "run".

Quoted 6 lines

On high-end systems sometimes a CPU is affined to a particular IRQ (say, network interface). Another situation is where you bind a RT task to a given CPU. In these situations, you want everything else to _not_ run on the CPUs. I.e., `run --bind=!1' (note its easy to generate the bitmask here too, by ANDing the inverse of the given CPU against -1).

I like it. I will add the "!" syntax to the next release of run. However, I suspect that this is a more error prone method of providing generic process shielding than the init method discussed above as processes that don't explicitly get biased can still find their way to the shielded CPU.

Quoted 5 lines

At any rate, what is needed most is to standardize on an interface for these scheduling mechanisms. I guess its just CPU affinity we have to go ... since not much progress was made of my (proc-based) method vs. Ingo's (syscall-based) method, at this point either of the two being merged would make me happy.

I will happily add support to "run" for Ingo's system calls if they get merged. However, many of the more powerful features of the "run" command currently require /proc for other reasons. For example, setting the CPU bias for all processes in a specified list of process groups, or setting the CPU bias for all processes owned by a specified list of users all require that I walk /proc to find matching processes.

Also, regarding the mpadvise(3) library service that I've proposed, commands like MPA_CPU_ACTIVE and MPA_PRC_GETRUN currently parse files in /proc to determine the list of active CPUs on the system and the CPU that a process is currently running on. Thus, until I can get all of the CPU-centric information that I need via system calls, adding Ingo's system calls doesn't help me too much (though they would allow "run" to do simple CPU biasing tasks on systems without /proc).

Quoted 1 line

I assume you have no problems with it ...

It's exactly what I needed. I've been using it for a few weeks without any problems.

Quoted 3 lines

I think I'd like to add the change that the CPUs reported correspond to the physical CPU number and not the logical value we derive. On x86 this won't make a difference, but its a proper method I suspect.

I don't have a strong opinion here, though note that in our proprietary RTOS our policy is to try to never expose physical values to the user unless they really, really need to know them. If you change the interface to physical values, I will probably abstract back to logical inside my code so the user still deals with logical (though I could add a --physical option to force the bias to be interpreted as physical values).

Take care,
Jason