Skip to contentSkip to conversation

dynsched - different cpu schedulers per cpuset

RFC Felix Obenhuber 3 participants 5 messages Nov 26, 2006 – Nov 30, 2006

lore
Share & tools
lore
#0 lore raw

Hej,

we're a student group witch is working on a research project concerning the ability to switch the cpu scheduler of the linux kernel at runtime.

We use Peter Williams Plugsched patch [1] to get an interface for the different scheduler implementations. Some month ago we started to modify the code to allow different scheduler running on each cpu on an SMP system. The cpu<->scheduler mapping is controlled via cpusets. Thus you can switch the scheduler for a cpuset containing multiple cpus and keep the rest untouched.

The project is hosted on Sourceforge [2] and the current patch applies against 2.6.18 patched with plugsched.

Threre are still lots of issues - especially the migration of tasks between cpus with different schedulers is quite buggy.

Switching the scheduler on non smp configured systems works fine (tested
x86)

Refer the project instruction site [3] for further information and usage / patch instructions.

We'd be quite happy, if someone could take a look at what we've done to gain some feedback/suggestions about the used techniques and implementation. Some changes are already queued.

The project documentation (description/benchmarks/usage/bugs) is in progress and will be completed in about 4 weeks.

Thanks a lot.

cheers,

Felix

[1] http://sourceforge.net/projects/cpuse/
[2] http://sourceforge.net/projects/dynsched/
[3] http://dynsched.sourceforge.net

Signature · 4 lines
Felix Obenhuber felixatobenhuber.de
www.obenhuber.de/felix
GPG: F696D489
Sat Nov 18 15:56:31 CET 2006
#1 lore raw
Quoted 3 lines · Felix

The cpu<->scheduler mapping is controlled via cpusets. Thus you can switch the scheduler for a cpuset containing multiple cpus and keep the rest untouched.

I don't have comments on the main focus of this work - schedulers are not my expertise.

I just noticed this lkml post because of my interest in cpusets.

You should take a look at the work of Paul Menage (added to the cc list), who is splitting the cpuset code into:
1) a generic "container" mechanism,
2) separate CPU and Memory "controllers", and
3) various other additional "controllers".

See Paul Menage's most recent patch proposal at:
http://lkml.org/lkml/2006/11/17/217
Subject: [PATCH 0/6] Multi-hierarchy Process Containers
Date: Fri, 17 Nov 2006 11:11:59 -0800

The container mechanism uses a virtual file system derived from the cpuset code to provide a file system style (hierarchical names and classic Unix style file and directory permissions) naming of a partitioning of the tasks on a system.

By partitioning here, I mean a division of the tasks into several subsets, aka partition elements, which are non-overlapping and covering.

That is, each task is in one and only one of the partition elements, these partitions elements are named by the directories in the container file system, and the regular files in the container file system provide per-element attributes.

Then kernel facilities that can be considered as providing attributes for and control of subsets of tasks is represented as a controller, and attached to such a container.

Your dynamic scheduler mechanisms appear (from what I can tell after a brief glance) to be a candidate for being such a controller.

The upshot of this is that, if your work should proceed and eventually be considered for inclusion in the kernel (I have --no-- idea if that would be a good idea, either for the purposes of your student group, or for the kernel itself) then it would likely (if Menage's work is accepted) need to be recast as a "controller" in Menage's terms, not as an extension to cpusets.

If Menage succeeds, that should not actually be that big of a change, either semantically, or in coding details.

Good luck.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#2 lore raw
Quoted 4 lines · pj

See Paul Menage's most recent patch proposal at:
http://lkml.org/lkml/2006/11/17/217
Subject: [PATCH 0/6] Multi-hierarchy Process Containers
Date: Fri, 17 Nov 2006 11:11:59 -0800

I'm behind the times. Paul Menage's most recent proposal is at:

    http://lkml.org/lkml/2006/11/23/95
    Subject: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
    Date:    Thu, 23 Nov 2006 04:08:48 -0800
Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#3 lore raw
In reply to Paul Jackson
Quoted 3 lines · Paul Jackson

Your dynamic scheduler mechanisms appear (from what I can tell after a brief glance) to be a candidate for being such a controller.

Possibly, if it was some kind of multi-level scheduler - i.e. a top-level scheduler picks which container to run, and then a configurable per-container scheduler picks a task from that container.

But (having glanced at the code even less than you) it sounded like it was intended to be a single level scheduler, configured on a per-cpu basis. In that case tying it to (exclusive) cpusets sounds like it might be more reasonable.

Paul.

#4 lore raw
Quoted 3 lines

Possibly, if it was some kind of multi-level scheduler - i.e. a top-level scheduler picks which container to run, and then a configurable per-container scheduler picks a task from that container.

What do you think about Plugsched? Peter Williams introduced a simple scheduler interface for scheduler testing purposes. It should be possible to integrate into your CPU Controller. The CPU Controller seems to be quite suited for that purpose, as far as I can appraise after reading some of your documentation.

Unfortunately, I didn't know about CKRM some months ago, when we decided to use cpusets. As I wrote, Dynsched is a student project at sunset and it's future is quite ambiguous. Maybe we'll find some time to take a look at.

Quoted 4 lines

But (having glanced at the code even less than you) it sounded like it was intended to be a single level scheduler, configured on a per-cpu basis. In that case tying it to (exclusive) cpusets sounds like it might be more reasonable.

You're right. We tie up the scheduler to cpus with some per_cpu functionality, and chose cpusets for management, cause we didn't want add another interface (e.g sysfs files for each cpu). Cpusets is quite adequate for our purpose, but there are lots of issues we're currently dealing with. Think about (not exclusive) subsets with different schedulers...

Felix