Skip to contentSkip to conversation

Revised CKRM release

RFC Shailabh Nagar 9 participants 21 messages Apr 29, 2004 – May 5, 2004

lore
Share & tools
lore
#0 lore raw

The Class-based Resource Management project is happy to release the first bits of a working prototype following a major revision of its interface and internal organization.

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

New in this release are the following:

1) A filesystem-based user interface, proposed by Rik van Riel, to replace the system call interface in the previous prototype.

2) A hierarchy of classes can now be created so that a class (created per user, say) can subdivide its share allocation among its children classes (created one per application type), independent of its peer classes (other users).

3) A newly introduced notion of a classtype which defines what kind of kernel objects are being grouped into a class for regulation and monitoring. Grouping tasks, into the taskclass classtype, is the most commonly expected use. The prototype also implements the socketclass classtype, useful for controlling groups of sockets.

4) Resource controllers are now explicitly associated with a classtype. The CPU memory and I/O controllers (not yet implemented) will operate on taskclasses while the multiple accept queue controller operates on socketclasses.

5) A functional socketaq network controller which regulates the number of accepted TCP connections for groups of listening sockets.

The newly implemented features have been described at some length in a document posted on lkml a while back and available at

http://ckrm.sourceforge.net/CKRMmergedAPI-d6.txt

A revised description and update of the project webpages is in progress. The patches will be posted individually and are described below. They are also available on http://ckrm.sf.net.

Comments/feedback welcome. If this looks interesting, please consider joining the [email protected] mailing list.

-- Hubertus Franke, Shailabh Nagar, Chandra Seetharaman, Vivek Kashyap

CKRM Patches overview
---------------------

All patches against 2.6.5.

00-core.ckrm-E12.patch:

Core code of ckrm which glues the interface (rcfs), resource controllers (rc's) and classification engines (ce's) into the framework.

01-rcfs.ckrm-E7.patch:

Resource control filesystem (rcfs) forming the user interface to CKRM.

02-taskclass.ckrm-E12.patch:

Creates the taskclass classtype for use by resource controllers which operate on groups of tasks. The CPU, memory and I/O resource controllers will operate on taskclasses when their rewrite/port to the new API is complete. The patch includes the rcfs interface to taskclasses.

03-numtasks.ckrm-E12.patch:

A simple resource controller that limits the number of tasks that can be forked within a taskclass. Implemented mainly to serve as a prototype for resource controller writers. Modifications to kernel/exit.c and kernel/fork.c which should strictly be part of this patch are included in the 00-core.ckrm-E12.patch.

04-socketclass.ckrm-E12.patch:

Creates the socketclass classtype, alongwith its rcfs interface, for use by resource controllers which operate on groups of sockets.

05-socketaq.ckrm-E12.patch:

A resource controller that controls the number of accepted TCP connections. It is CKRM's first real controller. Changes include modifications to the TCP stack to implement multiple accept queues.

rbce.ckrm-E12:

Two classification engines (CE) to assist in automatic classification of tasks and sockets. The first one, rbce, implements a rule-based classification engine which is generic enough for most users. The second, called crbce, is a variant of rbce which additionally provides information on significant kernel events (where a task/socket could get reclassified) to userspace as well as reports per-process wait times for cpu, memory, io etc. Such information can be used by user level tools to reclassify tasks to new classes, change class shares etc.

#1 lore raw
Quoted 10 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

I'd still love to see practical problems this thing is solving. It's a few thousand lines of code, not written to linux style guidelines, sometimes particularly obsfucated with callbacks all over the place.

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

author
#2 lore raw

Hi Christoph,

Quoted 12 lines · Christoph Hellwig
Quoted 10 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

Quoted 1 line

I'd still love to see practical problems this thing is solving.

We'd outlined three scenarios in our OLS'03 presentation
http://ckrm.sourceforge.net/documentation/ckrm-ols03-presentation.pdf

a) application server serving multiple requests from customers of varying importance. The app server dynamically spawns processes to handle each customers request. We need to group all the processes that are currently serving a low-importance "bronze" customer and ensure they don't take resources away from the group serving a "gold" customer. Important criteria: don't assume the processes spawned always serve the same customer (or customer type) ie. retain the flexibility for them being able to have a "gold" priority for some time and then revert to a "bronze" status.
--> needs processes to be classified into groups and regulated
based on some app-specific rules which cannot be predicted by the kernel in advance.

b) Desktop user doing a combination of activites with different priorities (for him/her), say:
compiling (lower) + listening to music (higher)

     --> needs music player to be given a higher share of all 
resources (cpu, mem, io) than the compile.
  disk backup (very low) + checking email (higher)
     --> needs io requests for email to be given a higher "share"

c) Multiple User-mode Linux instances running on a box (for virtual hosting). Each uml instance, serving a different type of consumer (say paying vs. nonpaying) needs a different level of service.

    --> need to define groups of processes which are spawned by the 

same uml instance

Besides, we have

d) department servers: multiple users logging in. Limit each user/login to a fixed share of cpu/mem/io. --> need to define groups of processes with same uid/gid or sharing
the same tty....

e) monitor how much load is being seen by a related group of applications on a Linux box (perhaps to decide whether they're better hosted on another box). --> needs to group processes by application group even when the
command names are arbitrary), should accomodate short-lived apps etc.

f) tcp connection requests for an http server are coming from sites with varying importance to the httpd owner. Serve some sites preferentially. --> needs incoming tcp connections to be accepted at differential
rates for groups of listening sockets formed using source ip/port.

Quoted 1 line

It's a few thousand lines of code, not written to linux style guidelines,

Guilty as charged :-( We will work to fix that until all are happy :-)

Quoted 1 line

sometimes particularly obsfucated with callbacks all over the place.

Not guilty ! Callbacks all over the place keep the various components independent - the resource controllers (which are/will be patches over the kernel schedulers), the classification engine module (which assists in automatic classification of processes/sockets into groups using rules; but is completely optional) and any code for new kinds of groupings (other than tasks and sockets) that may be found useful to control as a set in future.

This independence is a feature - it allows the controller code that is deemed acceptable to the corresponding scheduler maintainer to be integrated without being dependent on acceptance of other scheduler modifications.

Of course, the core and user interface (rcfs) have to be included, but they're not that large (subjective biased opinion of course, but seriously, if there are suggestions on how we can make it even leaner, we're open to ideas).

Quoted 4 lines

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

Trying to achieve the same goals using abstractions built on top of process-centric rlimits will not work for examples like a) or e).

Also, if we want to regulate resource consumption by groups of sockets or other types of kernel objects, the wheel would need to be reinvented.

We believe that CKRM addresses both of the above concerns.

Quoted 6 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to Christoph Hellwig
Quoted 3 lines · Christoph Hellwig

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#4 lore raw
Quoted 11 lines · Rik van Riel
Quoted 3 lines · Christoph Hellwig

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

....and provided the groups of processes that are sought to be regulated as a unit are relatively static.

Quoted 5 lines

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

In overhead, I presume you're including the overhead of running as many uml instances as expected number of classes. Not just the slowdown of applications because they're running under a uml instance (instead of running native) ?

I think UML is justified more from a fault-containment point of view (where overheads are a lower priority) than from a performance isolation viewpoint.

In any case, a 30% overhead would send a large batch of higher-end server admins running to get a stick to beat you with :-)

#5 lore raw
Quoted 8 lines · Shailabh Nagar
Quoted 3 lines · Rik van Riel

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

....and provided the groups of processes that are sought to be regulated as a unit are relatively static.

Good point, I hadn't thought of that one.

It works for most of the workloads I had in mind, but you're right that it's not good enough for eg. the university shell server.

Quoted 10 lines
Quoted 4 lines

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

In overhead, I presume you're including the overhead of running as many uml instances as expected number of classes. Not just the slowdown of applications because they're running under a uml instance (instead of running native) ?

Quoted 6 lines

I think UML is justified more from a fault-containment point of view (where overheads are a lower priority) than from a performance isolation viewpoint.

In any case, a 30% overhead would send a large batch of higher-end server admins running to get a stick to beat you with :-)

True enough, but from my pov the flip side is that merging the CKRM memory resource enforcement module has the potential of undoing lots of the performance tuning that was done to the VM in 2.6.

That could result in bad performance even for the people who aren't using workload management at all...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#6 lore raw
Quoted 28 lines · Rik van Riel
Quoted 8 lines · Shailabh Nagar
Quoted 3 lines · Rik van Riel

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

....and provided the groups of processes that are sought to be regulated as a unit are relatively static.

Good point, I hadn't thought of that one.

It works for most of the workloads I had in mind, but you're right that it's not good enough for eg. the university shell server.

Quoted 9 lines
Quoted 4 lines

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

In overhead, I presume you're including the overhead of running as many uml instances as expected number of classes. Not just the slowdown of applications because they're running under a uml instance (instead of running native) ?

I think UML is justified more from a fault-containment point of view (where overheads are a lower priority) than from a performance isolation viewpoint.

Quoted 7 lines
Quoted 1 line

In any case, a 30% overhead would send a large batch of higher-end server admins running to get a stick to beat you with :-)

True enough, but from my pov the flip side is that merging the CKRM memory resource enforcement module has the potential of undoing lots of the performance tuning that was done to the VM in 2.6.

Agreed - CKRM's memory controller logic needs major rework for it to be acceptable....but I'm sure you can do something about it, Rik ! :-)

The cpu and I/O controllers will also have to be reworked since we now have the hierarchical class requirement as well as lower and upper bounds for shares.

Quoted 3 lines

That could result in bad performance even for the people who aren't using workload management at all...

Even with the earlier logic, the hope was that if people are not using
workload management at all, then the only overhead they would see would be the extra indirection into "find next class to schedule" (in any controller) since there would be only one default class in the system. In the cpu case, this overhead had been shown to be as low as 1-2% but memory overhead had not been measured.

Keeping overheads low (or zero) for those who don't care to use CKRM functionality is a high-priority design goal. Keeping it proportional to number of classes (with more significant degradations seen if the number of hierarchy levels increase) comes next.

Also, will the 2.6 VM improvements continue to work as designed if multiple UML instances are running, each replicating a large memory user (like say a JVM or a database server) ? Taking the application server serving a number of different customers. If we have to replicate the app server for each customer class (one on each UML instance), the app server's memory needs would get added to the equation n times and the benefits of 2.6 VM tuning might be lost.

-- Shailabh

#7 lore raw
In reply to Rik van Riel
Quoted 18 lines · Rik van Riel
Quoted 6 lines · Shailabh Nagar

Rik van Riel wrote:

Quoted 8 lines
Quoted 5 lines

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

....and provided the groups of processes that are sought to be regulated as a unit are relatively static.

Quoted 8 lines

Good point, I hadn't thought of that one.

It works for most of the workloads I had in mind, but you're right that it's not good enough for eg. the university shell server.

Let's keep in mind that UML provides you a virtual machine, its an architecture.

"User-Mode Linux gives you a virtual machine that may have more hardware and software virtual resources than your actual, physical computer. Disk storage for the virtual machine is entirely contained inside a single file on your physical machine. " straight from the side.

It has its well deserved place, its great for fault protection isolation, but it does have performance hits.
Futhermore, there is no resource management associated with it. As we pointed out at last year's OLS. UML and CKRM would go great in concert, as CKRM could deliver the resource management underneath. AFAIK, UML tasks manifest themselves as tasks in the host, hence the UML image would provide naturally a class that could be assigned resources under CKRM.

One part that can not be done with virtual machines is the case of a shared middleware, where different classes of work are driving through a shared space.

UML and CKRM solve different issues and IMHO they nicely complement each other.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

Quoted 7 lines

In overhead, I presume you're including the overhead of running as many uml instances as expected number of classes. Not just the slowdown of applications because they're running under a uml instance (instead of running native) ?

I think UML is justified more from a fault-containment point of view (where overheads are a lower priority) than from a performance isolation viewpoint.

In any case, a 30% overhead would send a large batch of higher-end server admins running to get a stick to beat you with :-)

Quoted 10 lines

True enough, but from my pov the flip side is that merging the CKRM memory resource enforcement module has the potential of undoing lots of the performance tuning that was done to the VM in 2.6.

That could result in bad performance even for the people who aren't using workload management at all...

I am sure that can be worked out .. take a look at the history of NUMA support When initially NUMA support was put in it was done through hooks at various interfaces. Now its nicely embedded and abstracted out through macros in the memory space and similar stuff is happening in the scheduler area through the introduction of sched domains. We don't pay for NUMA or SMT if we are running on a single CPU system.

So I/we presume a similar progression will take place here. CKRM in its current state provides you the infrastructure in the kernel. The way it is written or intended, it should provide no overhead when not configured, neglible overhead when compiled in but not used and "minimal/acceptable" overhead when resource management is enabled.

-- Hubertus

#8 lore raw
In reply to Shailabh Nagar
Quoted 4 lines · [email protected] said:

In overhead, I presume you're including the overhead of running as many uml instances as expected number of classes. Not just the slowdown of applications because they're running under a uml instance (instead of running native) ?

My next major UML project is going to be porting it back into the kernel. By this, I mean calling the internal system calls rather than the libc wrappers. Doing this will give you an object that can be built directly into the kernel, or insmodded, and when it's started, will be an in-kernel UML instance.

People look at this as an overhead reduction thing. It will do that, and open up more opportunities for reducing overhead later, but I'm doing it to make UML something of a virtualization toolkit. I'm envisioning that an in-kernel UML can be stripped down to just the VM system, and processes inside it will be confined to using a maximum amount of memory in total, but unrestricted in every other way. Or it could be a VM system plus scheduler, in which case their access to CPU would be controlled as well as memory. Basically, my long-term goal is to allow UML to allow containment of any combination of resources.

Longer-term than that, I would like the in-kernel vs userspace containment choice to be independent of everything else, so you'd be able to decide how you want to confine your processes, and then decide whether the container should be in userspace or inside the kernel.

Quoted 4 lines · [email protected] said:
Quoted 2 lines

....and provided the groups of processes that are sought to be regulated as a unit are relatively static.

Good point, I hadn't thought of that one.

I'm starting to. With the in-kernel UML stuff I describe above, it doesn't seem too hard to move a process from the host scheduler to a UML scheduler, for example. Moving a process into a confined memory pool looks harder, but I can see how it might be done. The UML would trade pages with the host, getting the process's memory in exchange for giving up free pages. So, it looks possible that you could migrate processes around between containers.

Quoted 7 lines · [email protected] said:

Also, will the 2.6 VM improvements continue to work as designed if multiple UML instances are running, each replicating a large memory user (like say a JVM or a database server) ? Taking the application server serving a number of different customers. If we have to replicate the app server for each customer class (one on each UML instance), the app server's memory needs would get added to the equation n times and the benefits of 2.6 VM tuning might be lost.

Well, you get to share text, and data is split n ways instead of being n chunks in one server, so to a first approximation, it looks like a wash to me.

If the different customer classes are using the same data, then you might get some duplication, although it might be possible to eliminate it.

Jeff

author
#9 lore raw
In reply to Rik van Riel
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Christoph Hellwig

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

Quoted 9 lines

User Mode Linux could definitely be an option for implementing resource management, provided that the overhead can be kept low enough.

For these purposes, "low enough" could be as much as 30% overhead, since that would still allow people to grow the utilisation of their server from a typical 10-20% to as much as 40-50%.

http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html

has some numbers comparing native Linux to UML (and against the Xen virtual machine monitor) but its on a 2.4 kernel.

Jeff, do you have any numbers for UML overhead in 2.6 ?

-- Shailabh

#10 lore raw
Quoted 1 line · [email protected] said:

Jeff, do you have any numbers for UML overhead in 2.6 ?

It obviously depends on the workload, but for "normal" things, like kernel builds and web serving, it's generally in the 20-30% range. That can be reduced, since I haven't spent too much time on tuning. I'm aiming for the teens, and I don't think that'll be too hard.

Jeff

#11 lore raw
Quoted 7 lines · Jeff Dike
Quoted 1 line · [email protected] said:

Jeff, do you have any numbers for UML overhead in 2.6 ?

It obviously depends on the workload, but for "normal" things, like kernel builds and web serving, it's generally in the 20-30% range. That can be reduced, since I haven't spent too much time on tuning. I'm aiming for the teens, and I don't think that'll be too hard.

hmm, just wanted to mention that linux-vserver has around 0% overhead and often allows to improve performance due to resource sharing ...

basically it's a soft partitioning concept based on 'Security Contexts' which allow to create many independant Virtual Private Servers (VPS), which act simultaneously on one box at full speed, sharing the available hardware resources.

see http://linux-vserver.org for details ...

best,
Herbert

PS: UML and Linux-VServer play together nicely ...

Quoted 11 lines

Jeff

-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

#12 lore raw

В Сбт, 01.05.2004, в 02:43, Herbert Poetzl пишет:

Quoted 13 lines · Jeff Dike
Quoted 7 lines
Quoted 1 line · [email protected] said:

Jeff, do you have any numbers for UML overhead in 2.6 ?

It obviously depends on the workload, but for "normal" things, like kernel builds and web serving, it's generally in the 20-30% range. That can be reduced, since I haven't spent too much time on tuning. I'm aiming for the teens, and I don't think that'll be too hard.

hmm, just wanted to mention that linux-vserver has around 0% overhead and often allows to improve performance due to resource sharing ...

Herber please not say vserver have - 0 overhead.
it generally wrong.
But overhead less than UML is right.

Quoted 29 lines

basically it's a soft partitioning concept based on 'Security Contexts' which allow to create many independant Virtual Private Servers (VPS), which act simultaneously on one box at full speed, sharing the available hardware resources.

see http://linux-vserver.org for details ...

best,
Herbert

PS: UML and Linux-VServer play together nicely ...

Quoted 11 lines

Jeff

-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Alex Lyashkov <[email protected]>
PSoft
#13 lore raw
Quoted 10 lines · Alex Lyashkov

? ???, 01.05.2004, ? 02:43, Herbert Poetzl ?????:

Quoted 9 lines · Jeff Dike
Quoted 7 lines
Quoted 1 line · [email protected] said:

Jeff, do you have any numbers for UML overhead in 2.6 ?

It obviously depends on the workload, but for "normal" things, like kernel builds and web serving, it's generally in the 20-30% range. That can be reduced, since I haven't spent too much time on tuning. I'm aiming for the teens, and I don't think that'll be too hard.

Quoted 6 lines
Quoted 4 lines

hmm, just wanted to mention that linux-vserver has around 0% overhead and often allows to improve performance due to resource sharing ...

Herber please not say vserver have - 0 overhead.
it generally wrong.

well, I said around 0%, but it's actually a long time since we measured that, and I'll schedule some tests next week, to see if the overhead is still not measureable with 'normal' userspace testing

Quoted 1 line

But overhead less than UML is right.

that is for sure, and it benefits from not having everything twice, like inode cache, dentry cache, page cache ...

best,
Herbert

Quoted 32 lines
Quoted 29 lines

basically it's a soft partitioning concept based on 'Security Contexts' which allow to create many independant Virtual Private Servers (VPS), which act simultaneously on one box at full speed, sharing the available hardware resources.

see http://linux-vserver.org for details ...

best,
Herbert

PS: UML and Linux-VServer play together nicely ...

Quoted 11 lines

Jeff

-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Alex Lyashkov <[email protected]>
PSoft
#14 lore raw

В Сбт, 01.05.2004, в 17:46, Herbert Poetzl пишет:

Quoted 15 lines · Alex Lyashkov
Quoted 14 lines

? ???, 01.05.2004, ? 02:43, Herbert Poetzl ?????:

Quoted 13 lines · Jeff Dike
Quoted 7 lines
Quoted 1 line · [email protected] said:

Jeff, do you have any numbers for UML overhead in 2.6 ?

It obviously depends on the workload, but for "normal" things, like kernel builds and web serving, it's generally in the 20-30% range. That can be reduced, since I haven't spent too much time on tuning. I'm aiming for the teens, and I don't think that'll be too hard.

hmm, just wanted to mention that linux-vserver has around 0% overhead and often allows to improve performance due to resource sharing ...

Quoted 8 lines
Quoted 2 lines

Herber please not say vserver have - 0 overhead.
it generally wrong.

well, I said around 0%, but it's actually a long time since we measured that, and I'll schedule some tests next week, to see if the overhead is still not measureable with 'normal' userspace testing

Try compare system with 10 vsevers with with 1000 iptables rules per vserver and routing tables who has more one then one record :) and system without vservers with same setings ( 1000 iptables rules and some roting table). Other point who been slowly in vserver large sockets lists - try use 1000 simultaneous tcp connections per vserver ..... and other... :)
without iptables and 1-10 tcp connections per vserver you can`t find a speed decrease.

Also fix bugs with wrong selected source address for packets who send from vserver.

Quoted 47 lines
Quoted 2 lines

But overhead less than UML is right.

that is for sure, and it benefits from not having
everything twice, like inode cache, dentry cache,
page cache ...

best,
Herbert

Quoted 24 lines
Quoted 24 lines

basically it's a soft partitioning concept based on 'Security Contexts' which allow to create many independant Virtual Private Servers (VPS), which act simultaneously on one box at full speed, sharing the available hardware resources.

see http://linux-vserver.org for details ...

best,
Herbert

PS: UML and Linux-VServer play together nicely ...

Quoted 11 lines

Jeff

-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Quoted 8 lines
Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Alex Lyashkov <[email protected]>
PSoft

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Alex Lyashkov <[email protected]>
PSoft
#15 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

Quoted 7 lines

I'd still love to see practical problems this thing is solving. It's a few thousand lines of code, not written to linux style guidelines, sometimes particularly obsfucated with callbacks all over the place.

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

I have been reading CKRM docs this week and I think something which provides the same functionality is required for v2.7.

I haven't read the code yet, though. It probably should be converted to "linux style" and simplified whenever possible.

Right now our resource-limit infrastructure is very basic and limited. CKRM provides advanced/fine grained resource management.

#16 lore raw
Quoted 25 lines · Marcelo Tosatti
Quoted 12 lines · Christoph Hellwig
Quoted 12 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

Quoted 9 lines

I'd still love to see practical problems this thing is solving. It's a few thousand lines of code, not written to linux style guidelines, sometimes particularly obsfucated with callbacks all over the place.

I'd hate to see this in the kernel unless there's a very strong need for it and no way to solve it at a nicer layer of abstraction, e.g. userland virtual machines ala uml/umlinux.

Quoted 8 lines

I have been reading CKRM docs this week and I think something which provides the same functionality is required for v2.7.

I haven't read the code yet, though. It probably should be converted to "linux style" and simplified whenever possible.

Right now our resource-limit infrastructure is very basic and limited. CKRM provides advanced/fine grained resource management.

Marcelo, that was the intention, namely resource management within a single OS image for a variety of workload. The unit of resource management is a flexible resource classes. At the current time we have mainly focused on the infrastructure and have not spent the cycles to forward port our previous schedulers.
Those schedulers were mainly meant to be a starting discussion point to start thinking how these schedulers are to be written in a matter that they are acceptable to the mainline, i.e. no or minimal additional overhead, preserving existing heuristics, etc. etc.

As for your subsequent message on the classification engine.
The classification engine is optional. If configured it is also called along various kernel events, such as fork, exec, exit ... Rather than throwing this into userspace, we compromised on putting this into a loadable module.

The new RCFS (Resource Control FileSystem) brings a generic, expandable interface to the table. Let us know what other questions you have ....

-- Hubertus

#17 lore raw
In reply to Shailabh Nagar
Quoted 4 lines · Shailabh Nagar

The Class-based Resource Management project is happy to release the first bits of a working prototype following a major revision of its interface and internal organization.

Quoted 10 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

Cool!

Quoted 4 lines

New in this release are the following:

rbce.ckrm-E12:

Quoted 9 lines

Two classification engines (CE) to assist in automatic classification of tasks and sockets. The first one, rbce, implements a rule-based classification engine which is generic enough for most users. The second, called crbce, is a variant of rbce which additionally provides information on significant kernel events (where a task/socket could get reclassified) to userspace as well as reports per-process wait times for cpu, memory, io etc. Such information can be used by user level tools to reclassify tasks to new classes, change class shares etc.

It sounds to me the classification engine can be moved to userspace?

Such "classification" sounds a better suited to be done there.

Note: I haven't read the code yet.

#18 lore raw
Quoted 36 lines · Marcelo Tosatti
Quoted 4 lines · Shailabh Nagar

The Class-based Resource Management project is happy to release the first bits of a working prototype following a major revision of its interface and internal organization.

Quoted 10 lines

The basic concepts and motivation of CKRM remain the same as described in the overview at http://ckrm.sf.net. Privileged users can define classes consisting of groups of kernel objects (currently tasks and sockets) and specify shares for these classes. Resource controllers, which are independent of each other, can regulate and monitor the resources consumed by classes e.g the CPU controller will control the CPU time received by a class etc. Optional classification engines, implemented as kernel modules, can assist in the automatic classification of the kernel objects (tasks/sockets currently) into classes.

Cool!

Quoted 4 lines

New in this release are the following:

rbce.ckrm-E12:

Quoted 9 lines

Two classification engines (CE) to assist in automatic classification of tasks and sockets. The first one, rbce, implements a rule-based classification engine which is generic enough for most users. The second, called crbce, is a variant of rbce which additionally provides information on significant kernel events (where a task/socket could get reclassified) to userspace as well as reports per-process wait times for cpu, memory, io etc. Such information can be used by user level tools to reclassify tasks to new classes, change class shares etc.

Quoted 3 lines

It sounds to me the classification engine can be moved to userspace?

Such "classification" sounds a better suited to be done there.

I suppose it could. However, one of our design objectives was to support multi-threaded server apps where each thread (task) changes its class fairly rapidly (say every time it starts doing work on behalf of a more/less important transaction). Doing a transition to userspace and back may be too costly for such a scenario.

There might also be some concerns with keeping the reclassify operation atomic wrt deletion of the target class...but we haven't thought this through for userspace classification.

Quoted 3 lines

Note: I haven't read the code yet.

Why just read when you can test as well :-) We just released a testing tarball at http://ckrm.sf.net.. any inputs, bugs will be most welcome !

Looking forward to more inputs,
-- Shailabh

#19 lore raw
Quoted 9 lines · Shailabh Nagar
Quoted 3 lines

It sounds to me the classification engine can be moved to userspace?

Such "classification" sounds a better suited to be done there.

I suppose it could. However, one of our design objectives was to support multi-threaded server apps where each thread (task) changes its class fairly rapidly (say every time it starts doing work on behalf of a more/less important transaction). Doing a transition to userspace and back may be too costly for such a scenario.

But who sets the priority of the tasks is userspace anyway, isnt? AFAICS its userspace who knows which transaction is more/less important.

Quoted 3 lines

There might also be some concerns with keeping the reclassify operation atomic wrt deletion of the target class...but we haven't thought this through for userspace classification.

How often is a reclassify operation done?

Quoted 7 lines
Quoted 2 lines

Note: I haven't read the code yet.

Why just read when you can test as well :-) We just released a testing tarball at http://ckrm.sf.net.. any inputs, bugs will be most welcome !

Looking forward to more inputs,

Yeah, I'm just nitpicking from the outside and haven't contributed to anything, so...

#20 lore raw
Quoted 5 lines · Marcelo Tosatti
Quoted 5 lines
Quoted 4 lines

It sounds to me the classification engine can be moved to userspace?

Such "classification" sounds a better suited to be done there.

Quoted 7 lines
Quoted 5 lines

I suppose it could. However, one of our design objectives was to support multi-threaded server apps where each thread (task) changes its class fairly rapidly (say every time it starts doing work on behalf of a more/less important transaction). Doing a transition to userspace and back may be too costly for such a scenario.

But who sets the priority of the tasks is userspace anyway, isnt? AFAICS its

I didn't understand what do you mean by this (in this context).. can you elaborate ?

Quoted 1 line

userspace who knows which transaction is more/less important.

That is true, but the classfication event has to be notified to the user space from kernel, wight ? Shailabh is refering that overhead here.

Quoted 6 lines
Quoted 4 lines

There might also be some concerns with keeping the reclassify operation atomic wrt deletion of the target class...but we haven't thought this through for userspace classification.

How often is a reclassify operation done?

events like fork, exec, setuid/setgid etc., also when the application tag is explicitly changed(which could happen as many times as the application wants).

Quoted 23 lines
Quoted 8 lines
Quoted 3 lines

Note: I haven't read the code yet.

Why just read when you can test as well :-) We just released a testing tarball at http://ckrm.sf.net.. any inputs, bugs will be most welcome !

Looking forward to more inputs,

Yeah, I'm just nitpicking from the outside and haven't contributed to anything, so...

-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------