RFC: THE OFFLINE SCHEDULER
The Open University of Israel
Department of Mathematics and computer science
FINAL PAPER
OFFLINE SCHEDULER
OFFSCHED is a platform aimed to assign an assignment to an offloaded processor.An offloaded processor is a processor that is hot un-plugged from the operating system.
Description
In today’s computer world, we find that most processors have several embedded cores and hyper-threading. Most programmers do not really use these powerful features and let the operating system do the work.
At most, a programmer will bound an application to a certain processor or assign an interrupt to a different processor. At the end, we get system busy in maintaining tasks across processors, balancing interrupts, flushing TLBs and DTLBs using atomic operations even when not needed and worst of all, spin locks across processors in vein; and the more processors the merrier. I argue that in some cases, part of this behavior is due to fact the multiple core operating system is not service oriented but a system oriented. There is no easy way to assign a processor to do a distinct service, undisturbed, accurate, and fast as long as the processor is an active part of an operating system and still be a part of most of the operating system address space.
OFFSCHED Purpose
The purpose of the OFFSCHED is to create a platform for services. For example, assume a firewall is being attacked; the Linux operating system will generate endless number of interrupts and/or softirqs to analyze the traffic and throw out bad packets. This is on the expense of “good” packets. Have you ever tried to “ssh” to an attacked machine? Who protects the operating system ? What if we can simply do the packet analysis outside the operating system, without being interrupted ?
Why not assign a core to do only “firewalling”? Or just routing? Design a new type of Real Time system? Maybe assign it as an ultra accurate timer? Create a delaying service that does not just spin? Offload a TCP stack? perhaps a new type of a locking scheme? New type bottom-halves? Debug a running kernel through an offloaded processor? Maybe assign a GPU to do other things than just graphics? Amdahl Law teaches us that linear speed-up is not very feasible , so why not spare a processor to do certain tasks better?
Technologically speaking, I am referring to the Linux kernel ability to virtually hot unplug a (SMT) processor ;but instead of letting it wonder in endless “halts”, assign it a service.
OFFSCHED project can be found at :
http://sos-linux.svn.sourceforge.net/viewvc/sos-linux/OFFSCHED/
I have implemented:
1. OFFSCHED-RT. A different kind of a real time system.
2. OFFSCHED-TIMER. A 1us timer.
3. OFFSCHED-RTOP. Remote top. OFFSCHED throws top-like information to a nother machine, while the OFFSCHED machine is not accessible
4. OFFSCHED NAPI. Offloaded NAPI
5. OFFCSHED SECURED. A firewall that protects the operating system.
PATCHES
This email is followed by three patches. hopefully they will make the mainline.
patch 1: offsched core. this is actually a set of utilities
that enable a redirection of the cpu_down to offsched. offsched itself is a set of drivers.
patch 2:
A tiny patch to export cpu_up.
patch 3:
include files
There are several more patches,they serve special purpose . like napi or rtop.
Author: Raz Ben Yehuda
Quoted 15 lines · raz ben yehuda
The Open University of Israel
Department of Mathematics and computer scienceFINAL PAPER
OFFLINE SCHEDULEROFFSCHED is a platform aimed to assign an assignment to an offloaded processor.An offloaded processor is a processor that is hot un-plugged from the operating system.
Description
In today’s computer world, we find that most processors have several embedded cores and hyper-threading. Most programmers do not really use these powerful features and let the operating system do the work.
At most, a programmer will bound an application to a certain processor or assign an interrupt to a different processor. At the end, we get system busy in maintaining tasks across processors, balancing interrupts, flushing TLBs and DTLBs using atomic operations even when not needed and worst of all, spin locks across processors in vein; and the more processors the merrier. I argue that in some cases, part of this behavior is due to fact the multiple core operating system is not service oriented but a system oriented. There is no easy way to assign a processor to do a distinct service, undisturbed, accurate, and fast as long as the processor is an active part of an operating system and still be a part of most of the operating system address space.
Quoted 7 lines
OFFSCHED Purpose
The purpose of the OFFSCHED is to create a platform for services. For example, assume a firewall is being attacked; the Linux operating system will generate endless number of interrupts and/or softirqs to analyze the traffic and throw out bad packets. This is on the expense of “good” packets. Have you ever tried to “ssh” to an attacked machine? Who protects the operating system ? What if we can simply do the packet analysis outside the operating system, without being interrupted ?
Why not assign a core to do only “firewalling”? Or just routing? Design a new type of Real Time system? Maybe assign it as an ultra accurate timer? Create a delaying service that does not just spin? Offload a TCP stack? perhaps a new type of a locking scheme? New type bottom-halves? Debug a running kernel through an offloaded processor? Maybe assign a GPU to do other things than just graphics? Amdahl Law teaches us that linear speed-up is not very feasible , so why not spare a processor to do certain tasks better?
Technologically speaking, I am referring to the Linux kernel ability to virtually hot unplug a (SMT) processor ;but instead of letting it wonder in endless “halts”, assign it a service.
Seems to me this boils down to a different way to make a SW box in a HW box, which already exists. What does this provide that partitioning a box with csets and virtualization doesn't?
-Mike
Quoted 24 lines · Mike Galbraith
Quoted 15 lines · raz ben yehuda
The Open University of Israel
Department of Mathematics and computer scienceFINAL PAPER
OFFLINE SCHEDULEROFFSCHED is a platform aimed to assign an assignment to an offloaded processor.An offloaded processor is a processor that is hot un-plugged from the operating system.
Description
In today’s computer world, we find that most processors have several embedded cores and hyper-threading. Most programmers do not really use these powerful features and let the operating system do the work.
At most, a programmer will bound an application to a certain processor or assign an interrupt to a different processor. At the end, we get system busy in maintaining tasks across processors, balancing interrupts, flushing TLBs and DTLBs using atomic operations even when not needed and worst of all, spin locks across processors in vein; and the more processors the merrier. I argue that in some cases, part of this behavior is due to fact the multiple core operating system is not service oriented but a system oriented. There is no easy way to assign a processor to do a distinct service, undisturbed, accurate, and fast as long as the processor is an active part of an operating system and still be a part of most of the operating system address space.Quoted 7 lines
OFFSCHED Purpose
The purpose of the OFFSCHED is to create a platform for services. For example, assume a firewall is being attacked; the Linux operating system will generate endless number of interrupts and/or softirqs to analyze the traffic and throw out bad packets. This is on the expense of “good” packets. Have you ever tried to “ssh” to an attacked machine? Who protects the operating system ? What if we can simply do the packet analysis outside the operating system, without being interrupted ?
Why not assign a core to do only “firewalling”? Or just routing? Design a new type of Real Time system? Maybe assign it as an ultra accurate timer? Create a delaying service that does not just spin? Offload a TCP stack? perhaps a new type of a locking scheme? New type bottom-halves? Debug a running kernel through an offloaded processor? Maybe assign a GPU to do other things than just graphics? Amdahl Law teaches us that linear speed-up is not very feasible , so why not spare a processor to do certain tasks better?
Technologically speaking, I am referring to the Linux kernel ability to virtually hot unplug a (SMT) processor ;but instead of letting it wonder in endless “halts”, assign it a service.
Quoted 3 lines
Seems to me this boils down to a different way to make a SW box in a HW box, which already exists. What does this provide that partitioning a box with csets and virtualization doesn't?
OFFSCHED does not compete with cpu sets nor virtualization.it is different.
1. Neither virtuallization nor cpu sets provide hard real time. OFFSCHED does this with a little cost and no impact on the OS.OFFSCHED is not just accurate , it is also extremely fast,after all, it is NMI'ed processor.
2. OFFSCHED has a access to every piece of memory in the system. so it can act as a centry for the system, or use linux facilities. Also, the kernel can access OFFSCHED memory, it is the same address space.
3. OFFSCHED can improve the linux OS ( NAPI,OFFSCHED firewall,RTOP ), while a guest OS cannot.
4. cpu sets cannot replace softirqs and hardirqs. OFFSCHED can. cpu sets deals with kernel threads and user space threads. in OFFSCHED we use offlets.
5. cpu sets and virtualization are services provided by the kernel to the "system".who serves the kernel ? who protects the kernel ?
6. offlets gives the programmer full control over an entire processor. no preemption, no interrupts, no quiesce. you know what happens , and when it happens.
I have this hard real time system several years on my SMP/MC/SMT machines. It serves me well. The core of OFFSCHED patch was 4 lines. So,i simply compile a ***entirely regular*** linux bzImage and that's it. It did not mess with drivers, spinlocks, softirqs ..., OFFSCHED just directed the cpu_down to my own hard real time piece of code. The rest of the kernel remained the same.
Quoted 2 lines
-Mike
Quoted 5 lines · raz ben yehuda
Quoted 3 lines · Mike Galbraith
Seems to me this boils down to a different way to make a SW box in a HW box, which already exists. What does this provide that partitioning a box with csets and virtualization doesn't?
Quoted 7 lines
OFFSCHED does not compete with cpu sets nor virtualization.it is different.
1. Neither virtuallization nor cpu sets provide hard real time. OFFSCHED does this with a little cost and no impact on the OS.OFFSCHED is not just accurate , it is also extremely fast,after all, it is NMI'ed processor.
Why not? Why can't I run an RT kernel with an RTOS guest and let it do it's deadline management thing?
Quoted 3 lines
2. OFFSCHED has a access to every piece of memory in the system. so it can act as a centry for the system, or use linux facilities. Also, the kernel can access OFFSCHED memory, it is the same address space.
Hm. That appears to be a self negating argument.
Quoted 6 lines
3. OFFSCHED can improve the linux OS ( NAPI,OFFSCHED firewall,RTOP ), while a guest OS cannot.
4. cpu sets cannot replace softirqs and hardirqs. OFFSCHED can. cpu sets deals with kernel threads and user space threads. in OFFSCHED we use offlets.
Which still looks like OS-fu to me.
Quoted 2 lines
5. cpu sets and virtualization are services provided by the kernel to the "system".who serves the kernel ? who protects the kernel ?
If either one can diddle the others ram, they are in no way isolated or protected, so can't even defend against their own bugs.
What protects a hard RT deadline from VM pressure, memory bandwidth consumption etc etc? Looks to me like it's soft RT, because you can't control the external variables.
Quoted 3 lines
6. offlets gives the programmer full control over an entire processor. no preemption, no interrupts, no quiesce. you know what happens , and when it happens.
If I can route interrupts such that only say network interrupts are delivered to my cset/vm core, and the guest OS is a custom high speed low drag application, I just don't see much difference.
Quoted 6 lines
I have this hard real time system several years on my SMP/MC/SMT machines. It serves me well. The core of OFFSCHED patch was 4 lines. So,i simply compile a ***entirely regular*** linux bzImage and that's it. It did not mess with drivers, spinlocks, softirqs ..., OFFSCHED just directed the
cpu_downto my own hard real time piece of code. The rest of the kernel remained the same.
Aaaaanyway, I'm not saying it's not a useful thing to do, just saying I don't see any reason you can't get essentially the same result with what's in the kernel now.
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · raz ben yehuda
Quoted 3 lines · Mike Galbraith
Seems to me this boils down to a different way to make a SW box in a HW box, which already exists. What does this provide that partitioning a box with csets and virtualization doesn't?
OFFSCHED does not compete with cpu sets nor virtualization.it is different.
Quoted 7 lines
Quoted 4 lines
1. Neither virtuallization nor cpu sets provide hard real time. OFFSCHED does this with a little cost and no impact on the OS.OFFSCHED is not just accurate , it is also extremely fast,after all, it is NMI'ed processor.
Why not? Why can't I run an RT kernel with an RTOS guest and let it do it's deadline management thing?
Have you ever tested how long a single context switch cost ? can you run this system with a 1us accuracy ? you cannot.try ftrac'ing your system. the interrupt alone costs several hundreds nano seconds. By the time you will be reaching your code, the deadline will be nearly gone.
Quoted 5 lines
Quoted 3 lines
2. OFFSCHED has a access to every piece of memory in the system. so it can act as a centry for the system, or use linux facilities. Also, the kernel can access OFFSCHED memory, it is the same address space.
Hm. That appears to be a self negating argument.
correct. but I can receive packets over napi and transmit packets over hard_start_xmit much faster than any guest OS. I can disable interrupts and move to poll mode, thus helping the operating system. can a guest OS help linux?
Quoted 8 lines
Quoted 6 lines
3. OFFSCHED can improve the linux OS ( NAPI,OFFSCHED firewall,RTOP ), while a guest OS cannot.
4. cpu sets cannot replace softirqs and hardirqs. OFFSCHED can. cpu sets deals with kernel threads and user space threads. in OFFSCHED we use offlets.
Which still looks like OS-fu to me.
I do not understand this remark.
Quoted 5 lines
Quoted 2 lines
5. cpu sets and virtualization are services provided by the kernel to the "system".who serves the kernel ? who protects the kernel ?
If either one can diddle the others ram, they are in no way isolated or protected, so can't even defend against their own bugs.
correct. but the same applies for a hosting OS. as you said , it is a self negating argument. what if your system is attacked by a RT task that saturate all cpu time, you will not even be able to know what is wrong with your system. in OFFSCHED-RTOP I show that even when attacked by a malicious task, I still can see the problem because I can access the task list and dump it to a remote machine. It is even possible to "kill it" with the offlet-server ( still need to write the killing thing ).
Quoted 3 lines
What protects a hard RT deadline from VM pressure, memory bandwidth consumption etc etc? Looks to me like it's soft RT, because you can't control the external variables.
what does protect guest OS from the host ? also, In one of my applications I wrote my own pre-allocation system, OFFSCHED used only its own pools. so VM was never a problem and it is a true hard real time system. as for memory bandwidth pressure OFFSCHED is not protected from. But if you design your application to use small footprint, you will be able to stay in the processor cache. when you have a kernel thread, lazy TLB is not always promised. Can you say your RT task will never be preempted ?
And again, if RTAI or anything of the like has facilities for this kind of problems, OFFSCHED can use them as well.
Quoted 7 lines
Quoted 3 lines
6. offlets gives the programmer full control over an entire processor. no preemption, no interrupts, no quiesce. you know what happens , and when it happens.
If I can route interrupts such that only say network interrupts are delivered to my cset/vm core, and the guest OS is a custom high speed low drag application, I just don't see much difference.
There are other tasks a system must walk through , for example, a processor must walk through a quiesce state, which means you cannot have your real time thread running forever without loosing processor from time to time. and how would you prevent RCU starvation ? what about IPI ?
Quoted 10 lines
Quoted 6 lines
I have this hard real time system several years on my SMP/MC/SMT machines. It serves me well. The core of OFFSCHED patch was 4 lines. So,i simply compile a ***entirely regular*** linux bzImage and that's it. It did not mess with drivers, spinlocks, softirqs ..., OFFSCHED just directed the
cpu_downto my own hard real time piece of code. The rest of the kernel remained the same.Aaaaanyway, I'm not saying it's not a useful thing to do, just saying I don't see any reason you can't get essentially the same result with what's in the kernel now.
I thank you for your interest.
Quoted 2 lines
-Mike
Quoted 15 lines · raz ben yehuda
Quoted 14 lines · Mike Galbraith
Quoted 12 lines · raz ben yehuda
Quoted 3 lines · Mike Galbraith
Seems to me this boils down to a different way to make a SW box in a HW box, which already exists. What does this provide that partitioning a box with csets and virtualization doesn't?
OFFSCHED does not compete with cpu sets nor virtualization.it is different.
1. Neither virtuallization nor cpu sets provide hard real time. OFFSCHED does this with a little cost and no impact on the OS.OFFSCHED is not just accurate , it is also extremely fast,after all, it is NMI'ed processor.
Quoted 6 lines
Quoted 2 lines
Why not? Why can't I run an RT kernel with an RTOS guest and let it do it's deadline management thing?
Have you ever tested how long a single context switch cost ? can you run this system with a 1us accuracy ? you cannot.try ftrac'ing your system. the interrupt alone costs several hundreds nano seconds. By the time you will be reaching your code, the deadline will be nearly gone.
I've measured context switch cost many times. The point though, wasn't how tight a constraint may be, you maintained that realtime was out the window, and I didn't see any reason for that to be the case.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
2. OFFSCHED has a access to every piece of memory in the system. so it can act as a centry for the system, or use linux facilities. Also, the kernel can access OFFSCHED memory, it is the same address space.
Hm. That appears to be a self negating argument.
correct. but I can receive packets over napi and transmit packets over
hard_start_xmitmuch faster than any guest OS. I can disable interrupts and move to poll mode, thus helping the operating system. can a guest OS help linux?
Depends entirely on the job at hand. If the job is running a firewall in kernel mode, no it won't cut the mustard.
(no offense intended, but this all sounds like a great big kernel module to me, one which doesn't even taint the kernel)
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
3. OFFSCHED can improve the linux OS ( NAPI,OFFSCHED firewall,RTOP ), while a guest OS cannot.
4. cpu sets cannot replace softirqs and hardirqs. OFFSCHED can. cpu sets deals with kernel threads and user space threads. in OFFSCHED we use offlets.
Which still looks like OS-fu to me.
I do not understand this remark.
Whether it's offlet, tasklet, insert buzz-word of the day, it's thread of execution management, which I called OS-fu, ie one of those things that OSs do.
The rest, I'll leave off replying to, we're kinda splitting hairs. I don't see a big generic benefit to OFFSCHED or ilk, others do.
-Mike
Quoted 2 lines · Mike Galbraith
The rest, I'll leave off replying to, we're kinda splitting hairs. I don't see a big generic benefit to OFFSCHED or ilk, others do.
No we are not splitting hairs. OFFSCHED takes the OS noise (interrupts, timers, RCU, cacheline stealing etc etc) out of certain processors. You cannot run an undisturbed piece of software on the OS right now.
Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Mike Galbraith
The rest, I'll leave off replying to, we're kinda splitting hairs. I don't see a big generic benefit to OFFSCHED or ilk, others do.
No we are not splitting hairs. OFFSCHED takes the OS noise (interrupts, timers, RCU, cacheline stealing etc etc) out of certain processors. You cannot run an undisturbed piece of software on the OS right now.
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
-Mike
Quoted 5 lines · Mike Galbraith
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Mike Galbraith
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
If there were isolation, that would make it much less alien to _me_. Isolation would kinda destroy the reason it was written though. RT application/OS is injected into the network stack, which is kinda cool, but makes the hairs on my neck stand up.
-Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
There are other disturbances other than interrupts. attached is a first draft for the 11-th real time conference ,( If it would be accepted ).
tar zxvf offsched.tgz
cd paper
make
kpdf offsched.pdf.
thank you
raz
Quoted 20 lines · Mike Galbraith
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Mike Galbraith
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
If there were isolation, that would make it much less alien to _me_. Isolation would kinda destroy the reason it was written though. RT application/OS is injected into the network stack, which is kinda cool, but makes the hairs on my neck stand up.
-Mike
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Mike Galbraith
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Christoph Lameter
Quoted 5 lines · Mike Galbraith
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Quoted 5 lines
Quoted 2 lines
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Well there are the low latency requirements still. Those need to be addressed in some form. Some of these ideas here are a starting point.
Quoted 5 lines
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Ok so how would you go about avoiding the OS noise which motivated the patches for the Offline scheduler?
Quoted 8 lines · Peter Zijlstra
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Is it the whole concept of isolating one or more cpus from all normal kernel tasks that you don't like, or just this particular implementation?
I ask because I know of at least one project that would have used this capability had it been available. As it stands they have to live with the usual kernel threads running on the cpu that they're trying to dedicate to their app.
Chris
Quoted 11 lines · Chris Friesen
Quoted 8 lines · Peter Zijlstra
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Quoted 8 lines
Is it the whole concept of isolating one or more cpus from all normal kernel tasks that you don't like, or just this particular implementation?
I ask because I know of at least one project that would have used this capability had it been available. As it stands they have to live with the usual kernel threads running on the cpu that they're trying to dedicate to their app.
Its already possible to *almost* vacate a CPU except for a handful of kernel threads.
There are various hacks being distributed which also offload / suppress timer and rcu activity from specific CPUs.
Everything I have looked at has been hackish and racy, and no one using this is pusing any of it upstream.
OFFLINING solves the problem in a minimalist way, and only for tasks with very limited interaction with the Kernel.
In contrast however, almost all tasks with such limited Kernel interaction should be able to do fine under PREEMPT_RT after some cpuset work.
For those which absolutely cannot handle a handful of kernel threads sharing the CPU, the only option today is one or another form of hackery, and amongst those options, this would seem attractive by its mere simplicity.
But complete CPU isolation for user-space tasks still eludes.
Sven
Quoted 5 lines
Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Chris Friesen
Quoted 8 lines · Peter Zijlstra
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Quoted 7 lines
Is it the whole concept of isolating one or more cpus from all normal kernel tasks that you don't like, or just this particular implementation?
I ask because I know of at least one project that would have used this capability had it been available. As it stands they have to live with the usual kernel threads running on the cpu that they're trying to dedicate to their app.
Its the simple fact of going around the kernel instead of using the kernel.
Going around the kernel doesn't benefit anybody, least of all Linux.
So its the concept of running stuff on a CPU outside of Linux that I don't like. I mean, if you want that, go ahead and run RTLinux, RTAI, L4-Linux etc.. lots of special non-Linux hypervisor/exo-kernel like things around for you to run things outside Linux with.
Quoted 23 lines · Peter Zijlstra
Quoted 11 lines · Chris Friesen
Quoted 8 lines · Peter Zijlstra
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Quoted 7 lines
Is it the whole concept of isolating one or more cpus from all normal kernel tasks that you don't like, or just this particular implementation?
I ask because I know of at least one project that would have used this capability had it been available. As it stands they have to live with the usual kernel threads running on the cpu that they're trying to dedicate to their app.
Its the simple fact of going around the kernel instead of using the kernel.
Quoted 6 lines
Going around the kernel doesn't benefit anybody, least of all Linux.
So its the concept of running stuff on a CPU outside of Linux that I don't like. I mean, if you want that, go ahead and run RTLinux, RTAI, L4-Linux etc.. lots of special non-Linux hypervisor/exo-kernel like things around for you to run things outside Linux with.
Hello Peter, Hello All.
First , It a pleasure seeing that you take interest in OFFSCHED. So thank you.
To my opinion this a matter of defining what a system is. Queuing theory teaches us that a system is defined to be everything within the boundary of the computer, this includes, peripherals, processors, RAM , operating system, the distribution and so on.
The kernel is merely a part of the SYSTEM, it is not THE SYSTEM; and it is not a blasphemy to bypass it.The kernel is not the goal and it is not sacred.
OFFSCHED is bad name to my project. My project is called SOS = Service Oriented System.
SOS, has nothing to do with Real time. SOS is about arranging the processors to serve the SYSTEM the best way we can; if the kernel disturbs the service, put it a side I say.
How will the kernel is going to handle 32 processors machines ? These numbers are no longer a science-fiction.
What i am suggesting is merely a different approach of how to handle multiple core systems. instead of thinking in processes, threads and so on i am thinking in services. Why not take a processor and define this processor to do just firewalling ? encryption ? routing ? transmission ? video processing... and so on...
Raz
Quoted 3 lines · raz ben yehuda
OFFSCHED is bad name to my project. My project is called SOS = Service Oriented System.
SOS, has nothing to do with Real time.
??
The paper you pointed me at maintains it's very much about realtime.
<quote>
This paper argues that OFFSCHED fits to the niche of Multiprocessors real time systems by partitioning a system to two; the operating system and OFFSCHED. OFFSHCED is a hybrid system. It is hybrid because it is both real time and still a regular Linux server. Real time is mainly achieved by the NMI characteristic and the CPU isolation. It is a hybrid system because OFFSCHED scheduler interacts with the operating system. </quote>
-Mike.
Quoted 19 lines · Mike Galbraith
Quoted 3 lines · raz ben yehuda
OFFSCHED is bad name to my project. My project is called SOS = Service Oriented System.
SOS, has nothing to do with Real time.??
The paper you pointed me at maintains it's very much about realtime.
<quote>
This paper argues that OFFSCHED fits to the niche of Multiprocessors real time systems by partitioning a system to two; the operating system and OFFSCHED. OFFSHCED is a hybrid system. It is hybrid because it is both real time and still a regular Linux server. Real time is mainly achieved by the NMI characteristic and the CPU isolation. It is a hybrid system because OFFSCHED scheduler interacts with the operating system. </quote>
Quoted 3 lines
-Mike.
Mike Hello
Correct. OFFSCHED has a real time facet that puts it in the SMP real time system arena. it has other facets such as security and monitoring. If you take a look at OFFSCHED-RTOP and OFFSCHED-SECURED you will see that I can actually get information and change a system properties (very pool implementation so far, i am a bit tired...) even if this kernel is not **accessible**.
Raz
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · raz ben yehuda
How will the kernel is going to handle 32 processors machines ? These numbers are no longer a science-fiction.
The kernel is already running on 4096 processor machines. Dont worry about that.
Quoted 5 lines
What i am suggesting is merely a different approach of how to handle multiple core systems. instead of thinking in processes, threads and so on i am thinking in services. Why not take a processor and define this processor to do just firewalling ? encryption ? routing ? transmission ? video processing... and so on...
I think that is a valuable avenue to explore. What we do so far is treating each processor equally. Dedicating a processor has benefits in terms of cache hotness and limits OS noise.
Most of the large processor configurations already partition the system using cpusets in order to limit the disturbance by OS processing. A set of cpus is used for OS activities and system daemons are put into that set. But what can be done is limited because the OS threads as well as interrupt and timer processing etc cannot currently be moved. The ideas that you are proposing are particularly usedful for applications that require low latencies and cannot tolerate OS noise easily (Infiniband MPI base jobs f.e.)
Quoted 18 lines · Christoph Lameter
Quoted 2 lines · raz ben yehuda
How will the kernel is going to handle 32 processors machines ? These numbers are no longer a science-fiction.
The kernel is already running on 4096 processor machines. Dont worry about that.
Quoted 5 lines
What i am suggesting is merely a different approach of how to handle multiple core systems. instead of thinking in processes, threads and so on i am thinking in services. Why not take a processor and define this processor to do just firewalling ? encryption ? routing ? transmission ? video processing... and so on...
I think that is a valuable avenue to explore. What we do so far is treating each processor equally. Dedicating a processor has benefits in terms of cache hotness and limits OS noise.
Quoted 8 lines
Most of the large processor configurations already partition the system using cpusets in order to limit the disturbance by OS processing. A set of cpus is used for OS activities and system daemons are put into that set. But what can be done is limited because the OS threads as well as interrupt and timer processing etc cannot currently be moved. The ideas that you are proposing are particularly usedful for applications that require low latencies and cannot tolerate OS noise easily (Infiniband MPI base jobs f.e.)
My 0.2 cents:
I have always been fascinated by the idea of controlling another cpu from the main CPU.
Usually these cpus are custom, run proprietary software, and have no datasheet on their I/O interfaces.
But, being able to turn an ordinary CPU into something like that seems to be very nice.
For example, It might help with profiling. Think about a program that can run uninterrupted how much it wants.
I might even be better, if the dedicated CPU would use a predefined reserved memory range (I wish there was a way to actually lock it to that range)
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
In the other words, we might see 'firmwares' that take an entire cpu for their usage.
Quoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 46 lines · Maxim Levitsky
Quoted 18 lines · Christoph Lameter
Quoted 2 lines · raz ben yehuda
How will the kernel is going to handle 32 processors machines ? These numbers are no longer a science-fiction.
The kernel is already running on 4096 processor machines. Dont worry about that.
Quoted 5 lines
What i am suggesting is merely a different approach of how to handle multiple core systems. instead of thinking in processes, threads and so on i am thinking in services. Why not take a processor and define this processor to do just firewalling ? encryption ? routing ? transmission ? video processing... and so on...
I think that is a valuable avenue to explore. What we do so far is treating each processor equally. Dedicating a processor has benefits in terms of cache hotness and limits OS noise.
Quoted 8 lines
Most of the large processor configurations already partition the system using cpusets in order to limit the disturbance by OS processing. A set of cpus is used for OS activities and system daemons are put into that set. But what can be done is limited because the OS threads as well as interrupt and timer processing etc cannot currently be moved. The ideas that you are proposing are particularly usedful for applications that require low latencies and cannot tolerate OS noise easily (Infiniband MPI base jobs f.e.)
My 0.2 cents:
I have always been fascinated by the idea of controlling another cpu from the main CPU.
Usually these cpus are custom, run proprietary software, and have no datasheet on their I/O interfaces.
But, being able to turn an ordinary CPU into something like that seems to be very nice.
For example, It might help with profiling. Think about a program that can run uninterrupted how much it wants.
I might even be better, if the dedicated CPU would use a predefined reserved memory range (I wish there was a way to actually lock it to that range)
Quoted 7 lines
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
In the other words, we might see 'firmwares' that take an entire cpu for their usage.
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU. Processors are to become more and more redundant and Linux as an
evolutionary system must use it. why not offload raid5 write engine ?
why not encrypt in a different processor ?
Also , having so many processors in a single OS means a bug prone
system , with endless contention points when two or more OS processors
interacts. let's make things simpler.
Quoted 8 lines
Quoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 21 lines · raz ben yehuda
Quoted 21 lines
I have always been fascinated by the idea of controlling another cpu from the main CPU.
Usually these cpus are custom, run proprietary software, and have no datasheet on their I/O interfaces.
But, being able to turn an ordinary CPU into something like that seems to be very nice.
For example, It might help with profiling. Think about a program that can run uninterrupted how much it wants.
I might even be better, if the dedicated CPU would use a predefined reserved memory range (I wish there was a way to actually lock it to that range)
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
Quoted 4 lines
Quoted 2 lines
In the other words, we might see 'firmwares' that take an entire cpu for their usage.
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU.
So where are the patches? The URL in the original post returns 404...
sos linux is at:
http://sos-linux.svn.sourceforge.net/viewvc/sos-linux/offsched/
you will find the modules, once shot patches , split patches, and a Documentation folder.
Quoted 28 lines · Pekka Enberg
Quoted 21 lines · raz ben yehuda
Quoted 21 lines
I have always been fascinated by the idea of controlling another cpu from the main CPU.
Usually these cpus are custom, run proprietary software, and have no datasheet on their I/O interfaces.
But, being able to turn an ordinary CPU into something like that seems to be very nice.
For example, It might help with profiling. Think about a program that can run uninterrupted how much it wants.
I might even be better, if the dedicated CPU would use a predefined reserved memory range (I wish there was a way to actually lock it to that range)
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
Quoted 4 lines
Quoted 2 lines
In the other words, we might see 'firmwares' that take an entire cpu for their usage.
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU.
So where are the patches? The URL in the original post returns 404...
Quoted 2 lines · raz ben yehuda
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU.
It is not, GPUs and other paired chips form a hybrid system. Linux is known to run on one or more of such chips and communicate through whatever means these chips have.
But what you propose here is hard partitioning a homogeneous system, totally different.
Quoted 3 lines
Processors are to become more and more redundant and Linux as an evolutionary system must use it. why not offload raid5 write engine ? why not encrypt in a different processor ?
Why waste a whole cpu for something that could be done by part of one?
Quoted 3 lines
Also , having so many processors in a single OS means a bug prone system , with endless contention points when two or more OS processors interacts.
You're bound to have interaction between the core os and these partitions you want, non of it different from how threads in the kernel would interact, other than that you're going to re-invent everything already present in the kernel.
Quoted 1 line
let's make things simpler.
You don't, you make things more complex by introducing duplicate functionality.
What's more, you burden the user with having to configure such a system, and make choices on having to give up parts of his system, nothing like that should be needed on a homogeneous system.
Work spend on trimming fat of of the core kernel helps everybody, even users not otherwise interested in things like giving up a whole cpu for some odd purpose.
There is no reason something could be done more efficiently on a dedicated CPU than not when you assume a homogeneous system (which is all Linux supports in the single image sense).
If you think the kernel is too fat and does superfluous things for your needs, help trim it.
Quoted 1 line · Peter Zijlstra
Why waste a whole cpu for something that could be done by part of one?
Because of latency and performance requirements
Quoted 4 lines
You're bound to have interaction between the core os and these partitions you want, non of it different from how threads in the kernel would interact, other than that you're going to re-invent everything already present in the kernel.
The kernel interactions can be done while running on another (not isolated) cpu.
Quoted 2 lines
You don't, you make things more complex by introducing duplicate functionality.
The functionality does not exist. This is about new features.
Quoted 2 lines
If you think the kernel is too fat and does superfluous things for your needs, help trim it.
Mind boogling nonsense. Please stop fantasizing and trolling.
Quoted 5 lines · Christoph Lameter
Quoted 1 line · Peter Zijlstra
Why waste a whole cpu for something that could be done by part of one?
Because of latency and performance requirements
Latency is the only one, and yes people have been using hacks like this, I've also earlier mentioned RTAI, RTLinux and L4-Linux which basically do the same thing.
The problem is, that its not linux, you cannot run something on a these off-cores and use the same functionality as linux, if you could it'd not be offline.
The past year or so you've been whining about the tick latency, and I've seen exactly _0_ patches from you slimming down the work done in there, even though I pointed out some obvious things that could be done.
Carving out cpus just doesn't work in the long run (see below for more), it adds configuration burdens on people and it would duplicate functionality (below), or it provides it in a (near) useless manner.
If you were to work on lowering the linux latency in the full kernel sense, you'd help out a lot of people, many use-cases would improve and you'd be helpful to he greater good.
If you hack up special cases like this, then only your one use-case gets better and the rest doesn't, or it might actually get worse, because it got less attention.
Quoted 7 lines
Quoted 4 lines
You're bound to have interaction between the core os and these partitions you want, non of it different from how threads in the kernel would interact, other than that you're going to re-invent everything already present in the kernel.
The kernel interactions can be done while running on another (not isolated) cpu.
There needs to be some communication between the isolated and non isolated part, otherwise what's the point. Even when you'd let it handle say a network device as pure firewall, you'd need to configure the thing, requiring interaction.
Interaction of any sorts gets serialization requirements, and from there on things tend to grow.
Quoted 4 lines
Quoted 2 lines
You don't, you make things more complex by introducing duplicate functionality.
The functionality does not exist. This is about new features.
It is not, he is proposing to use these cores for:
- network stuff, we already have that
- raid5 stuff, we already have that
- other stuff we already have
Then there is the issue of what happens when a single core isn't sufficient for the given task, then you'd need to split up, again creating more interaction.
Quoted 4 lines
Quoted 2 lines
If you think the kernel is too fat and does superfluous things for your needs, help trim it.
Mind boogling nonsense. Please stop fantasizing and trolling.
Oh, to lay down the crack-pipe and sod off.
Hi Peter,
Quoted 4 lines · Peter Zijlstra
There needs to be some communication between the isolated and non isolated part, otherwise what's the point. Even when you'd let it handle say a network device as pure firewall, you'd need to configure the thing, requiring interaction.
The use case Christoph described was an user-space number cruncher app that does some network I/O over RDMA IIRC. AFAICT, if he could isolate a physical CPU for the thing, there would be little or no communication with the non-isolated part. Yes, the setup sounds weird but it's a real workload although pretty damn specialized.
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines
Quoted 2 lines
If you think the kernel is too fat and does superfluous things for your needs, help trim it.
Mind boogling nonsense. Please stop fantasizing and trolling.
Oh, to lay down the crack-pipe and sod off.
I guess I'll go for the magic mushrooms then.
Pekka
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Christoph Lameter
Quoted 1 line · Peter Zijlstra
Why waste a whole cpu for something that could be done by part of one?
Because of latency and performance requirements
Quoted 7 lines
Latency is the only one, and yes people have been using hacks like this, I've also earlier mentioned RTAI, RTLinux and L4-Linux which basically do the same thing.
The problem is, that its not linux, you cannot run something on a these off-cores and use the same functionality as linux, if you could it'd not be offline.
Right. We discussed this. Why are you repeating the same old arguments?
Quoted 3 lines
Carving out cpus just doesn't work in the long run (see below for more), it adds configuration burdens on people and it would duplicate functionality (below), or it provides it in a (near) useless manner.
Its pretty simple. Just isolate the cpu, forbid the OS to run anything on it. Allow a user space process to change its affinity to the isolated cpu. Should the process be so stupid as to ask the OS for services then just switch it back to a regular processor. Interaction is still possible via shared memory communication as well as memory mapped devices.
Quoted 3 lines
If you hack up special cases like this, then only your one use-case gets better and the rest doesn't, or it might actually get worse, because it got less attention.
What special case? This is a generic mechanism.
Quoted 7 lines
Quoted 2 lines
The kernel interactions can be done while running on another (not isolated) cpu.
There needs to be some communication between the isolated and non isolated part, otherwise what's the point. Even when you'd let it handle say a network device as pure firewall, you'd need to configure the thing, requiring interaction.
Shared memory, memory mapped devices?
Quoted 2 lines
Interaction of any sorts gets serialization requirements, and from there on things tend to grow.
Yes and there are mechanism that provide the serialization without OS services.
Quoted 7 lines
Quoted 1 line
The functionality does not exist. This is about new features.
It is not, he is proposing to use these cores for:
- network stuff, we already have that
- raid5 stuff, we already have that
- other stuff we already have
Right. I also want to use it for network stuff. Infiniband which support memory mapped registers and stuff. Its generic not special as you state.
Quoted 3 lines
Then there is the issue of what happens when a single core isn't sufficient for the given task, then you'd need to split up, again creating more interaction.
Well yes you need to create synchronization methods that do not require OS interaction.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If you think the kernel is too fat and does superfluous things for your needs, help trim it.
Mind boogling nonsense. Please stop fantasizing and trolling.
Oh, to lay down the crack-pipe and sod off.
Dont have one here. Whats a sod off?
Quoted 13 lines · Christoph Lameter
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Christoph Lameter
Quoted 1 line · Peter Zijlstra
Why waste a whole cpu for something that could be done by part of one?
Because of latency and performance requirements
Latency is the only one, and yes people have been using hacks like this, I've also earlier mentioned RTAI, RTLinux and L4-Linux which basically do the same thing.
Quoted 6 lines
Quoted 3 lines
The problem is, that its not linux, you cannot run something on a these off-cores and use the same functionality as linux, if you could it'd not be offline.
Right. We discussed this. Why are you repeating the same old arguments?
The thing is, you have cut out (and have not replied to) this
Quoted 4 lines · crutial bit of what Peter
Quoted 4 lines
The past year or so you've been whining about the tick latency, and I've seen exactly _0_ patches from you slimming down the work done in there, even though I pointed out some obvious things that could be done.
... which pretty much settles the issue as far as i'm concerned. If you were truly interested in a constructive solution to lower latencies in Linux you should have sent patches already for the low hanging fruits Peter pointed out.
Ingo
Quoted 8 lines · Ingo Molnar
The thing is, you have cut out (and have not replied to) this
Quoted 4 lines · crutial bit of what Peter
Quoted 4 lines
The past year or so you've been whining about the tick latency, and I've seen exactly _0_ patches from you slimming down the work done in there, even though I pointed out some obvious things that could be done.
Quoted 4 lines
... which pretty much settles the issue as far as i'm concerned. If you were truly interested in a constructive solution to lower latencies in Linux you should have sent patches already for the low hanging fruits Peter pointed out.
The noise latencies were already reduced in years earlier to the mininum (f.e. the work on slab queue cleaning). Certainly more could be done there but that misses the point.
The point of the OFFLINE scheduler is to completely eliminate the OS disturbances by getting rid of *all* OS processing on some cpus.
For some reason scheduler developers seem to be threatened by this idea and they go into bizarre lines of arguments to avoid the issue. Its simple and doable and the scheduler will still be there after we do this.
Quoted 10 lines · Christoph Lameter
Quoted 8 lines · Ingo Molnar
The thing is, you have cut out (and have not replied to) this
Quoted 4 lines · crutial bit of what Peter
Quoted 4 lines
The past year or so you've been whining about the tick latency, and I've seen exactly _0_ patches from you slimming down the work done in there, even though I pointed out some obvious things that could be done.
Quoted 8 lines
Quoted 4 lines
... which pretty much settles the issue as far as i'm concerned. If you were truly interested in a constructive solution to lower latencies in Linux you should have sent patches already for the low hanging fruits Peter pointed out.
The noise latencies were already reduced in years earlier to the mininum (f.e. the work on slab queue cleaning). Certainly more could be done there but that misses the point.
Peter suggested various improvements to the timer tick related latencies _you_ were complaining about earlier this year. Those latencies sure were not addressed 'years earlier'.
If you are unwilling to reduce the very latencies you apparently cared and complained about then you dont have much real standing to complain now.
( If you on the other hand were approaching this issue with
pragmatism and with intellectual honesty, if you were at the end of a string of patches that gradually improved latencies but couldnt get them below a certain threshold, and if scheduler developers couldnt give you any ideas what else to improve, and _then_ suggested some other solution, you might have a point. You are far away from being able to claim that. )
Really, it's a straightforward application of Occam's Razor to the scheduler. We go for the simplest solution first, and try to help more people first, before going for some specialist hack.
Quoted 8 lines
The point of the OFFLINE scheduler is to completely eliminate the OS disturbances by getting rid of *all* OS processing on some cpus.
For some reason scheduler developers seem to be threatened by this idea and they go into bizarre lines of arguments to avoid the issue. Its simple and doable and the scheduler will still be there after we do this.
If you meant to include me in that summary categorization, i dont feel 'threatened' by any such patches (why would i? They dont seem to have sharp teeth nor any apparent poison fangs) - i simply concur with the reasons Peter listed that it is a technically inferior solution.
Ingo
Quoted 7 lines · Ingo Molnar
( If you on the other hand were approaching this issue with
pragmatism and with intellectual honesty, if you were at the end of a string of patches that gradually improved latencies but couldnt get them below a certain threshold, and if scheduler developers couldnt give you any ideas what else to improve, and _then_ suggested some other solution, you might have a point. You are far away from being able to claim that. )
Intellectual honesty? Wish I would be seeing it. So far there is not even the uptake required on your side to discuss the problem.
There is no threshold. HPC and other industries want processors as a whole with all their abilities. They will squeeze the last bit of performance out of them.
Quoted 3 lines
to have sharp teeth nor any apparent poison fangs) - i simply concur with the reasons Peter listed that it is a technically inferior solution.
Ok so you are saying that the reduction of OS latencies will make the processor completely available and have no disturbances like OFFLINE scheduling?
Peter has not given a solution to the problem. Nor have you.
Quoted 1 line · On Wed, 26 Aug 2009 16:40:09 -0400 (EDT) Christoph Lameter…
Peter has not given a solution to the problem. Nor have you.
What problem?
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
What is the problem statement?
My take on these patches: the kernel gives userspace unmediated access to memory resources if it wants that. The kernel gives userspace unmediated access to IO devices if it wants that. But for some reason people freak out at the thought of providing unmediated access to CPU resources.
Don't get all religious about this. If the change is clean, maintainable and useful then there's no reason to not merge it.
Quoted 2 lines · Andrew Morton
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
Maybe. But its a problem statement that I have seen in various industries. Multiple kernel hacks exist to do this in more or less contorted way. We already have Linux scheduler functionality that does partially what is needed.
See the
isolcpus
kernel parameter. isolcpus does not switch off OS sources of noise but it takes the processor away from the scheduler. We need a harder form of isolation where the excluded processors offer no OS services at all.
Quoted 1 line
What is the problem statement?
My definition (likely not covering all that the author of this patchset wants):
How to make a processor in a multicore system completely available to a process.
Quoted 6 lines · Andrew Morton
What problem?
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
What is the problem statement?
I can only speak for myself...
In our case the problem statement was that we had an inherently single-threaded emulator app that we wanted to push as hard as absolutely possible.
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Chris
Quoted 15 lines · Chris Friesen
Quoted 6 lines · Andrew Morton
What problem?
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
What is the problem statement?
I can only speak for myself...
In our case the problem statement was that we had an inherently single-threaded emulator app that we wanted to push as hard as absolutely possible.
Quoted 7 lines
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Chris
Chris. I offer myself to help anyone wishes to apply OFFSCHED.
Quoted 10 lines · raz ben yehuda
Quoted 5 lines · Chris Friesen
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Chris. I offer myself to help anyone wishes to apply OFFSCHED.
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
Chris
Quoted 3 lines · Chris Friesen
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
It needs to be implemented the right way. Essentially this is a variation on the isolcpu kernel boot option. We probably need some syscall to move a user space process to a bare metal cpu since the cpu cannot be considered online in the regular sense.
An isolated cpu can then only execute one process at a time. A process would do all initialization and lock itsresources in memory before going to the isolated processor. Any attempt to use OS facilities need to cause the process to be moved back to a cpu with OS services.
Quoted 10 lines · Christoph Lameter
Quoted 3 lines · Chris Friesen
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
It needs to be implemented the right way. Essentially this is a variation on the isolcpu kernel boot option. We probably need some syscall to move a user space process to a bare metal cpu since the cpu cannot be considered online in the regular sense.
It can. It needs to be flagged as reserved for special tasks and you need a separate mechanism to move and pin a task to such a CPU.
Quoted 4 lines
An isolated cpu can then only execute one process at a time. A process would do all initialization and lock itsresources in memory before going to the isolated processor. Any attempt to use OS facilities need to cause the process to be moved back to a cpu with OS services.
You are creating a "one special case" operation mode which is not justified in my opinion. Let's look at the problem you want to solve:
Run exactly one thread on a dedicated CPU w/o any disturbance by the scheduler tick.
You can move away anything else than the scheduler tick from a CPU today already w/o a single line of code change.
But you want to impose restrictions like resource locking and moving back to another CPU in case of a syscall. What's the purpose of this ? It does not buy anything except additional complexity.
That's just the wrong approach. All you need is a way to tell the kernel that CPUx can switch off the scheduler tick when only one thread is running and that very thread is running in user space. Once another thread arrives on that CPU or the single thread enters the kernel for a blocking syscall the scheduler tick has to be restarted.
It's not rocket science to fix the well known issues of stopping and eventually restarting the scheduler tick, the CPU time accounting and some other small details. Such a modification would be of general use contrary to your proposed solution which is just a hack to solve your particular special case of operation.
Thanks,
tglx
Quoted 39 lines · Thomas Gleixner
Quoted 9 lines · Christoph Lameter
Quoted 3 lines · Chris Friesen
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
It needs to be implemented the right way. Essentially this is a variation on the isolcpu kernel boot option. We probably need some syscall to move a user space process to a bare metal cpu since the cpu cannot be considered online in the regular sense.
It can. It needs to be flagged as reserved for special tasks and you need a separate mechanism to move and pin a task to such a CPU.
Quoted 4 lines
An isolated cpu can then only execute one process at a time. A process would do all initialization and lock itsresources in memory before going to the isolated processor. Any attempt to use OS facilities need to cause the process to be moved back to a cpu with OS services.
You are creating a "one special case" operation mode which is not justified in my opinion. Let's look at the problem you want to solve:
Run exactly one thread on a dedicated CPU w/o any disturbance by the scheduler tick.
You can move away anything else than the scheduler tick from a CPU today already w/o a single line of code change.
But you want to impose restrictions like resource locking and moving back to another CPU in case of a syscall. What's the purpose of this ? It does not buy anything except additional complexity.
That's just the wrong approach. All you need is a way to tell the kernel that CPUx can switch off the scheduler tick when only one thread is running and that very thread is running in user space. Once another thread arrives on that CPU or the single thread enters the kernel for a blocking syscall the scheduler tick has to be restarted.
Quoted 5 lines
It's not rocket science to fix the well known issues of stopping and eventually restarting the scheduler tick, the CPU time accounting and some other small details. Such a modification would be of general use contrary to your proposed solution which is just a hack to solve your particular special case of operation.
I wonder if it makes sense to do something along the lines of the sched-class...
IOW: What if we adopted one of the following models:
1) Create a new class that is higher prio than FIFO/RR and, when selected, disables the tick.
2) Modify FIFO so that it disables tick by default...update accounting info at next reschedule event.
3) Variation of 2..leave FIFO+tick as is by default, but have some kind of parameter to optionally disable tick if desired.
In a way, we should probably consider (2) independent of this particular thread. FIFO doesn't need a tick anyway afaict...only a RESCHED+IPI truly ever matter here....or am I missing something obvious (probably w.r.t accounting)?
You could then couple this solution with cpusets (possibly with a little work to get rid of any pesky per-cpy kthreads) to achieve the desired effect of interference-free operation. You wouldn't even have to have funky rules eluded to above w.r.t. making sure only one userspace thread is running on the core.
Thoughts?
-Greg
Quoted 2 lines · Gregory Haskins
2) Modify FIFO so that it disables tick by default...update accounting info at next reschedule event.
I like it. The only thing to watch out for is that events that wake up higher-priority FIFO tasks do not get deferred :)
Signature · 1 line
All rights reversed.
Hi Rik,
Quoted 9 lines · Rik van Riel
Quoted 2 lines · Gregory Haskins
2) Modify FIFO so that it disables tick by default...update accounting info at next reschedule event.
I like it. The only thing to watch out for is that events that wake up higher-priority FIFO tasks do not get deferred :)
Yeah, agreed. My (potentially half-baked) proposal should work at least from a pure scheduling perspective since FIFO technically does not reschedule based on a tick, and wakeups/migrations should still work bidirectionally with existing scheduler policies.
However, and to what I believe is your point: its not entirely clear to me what impact, if any, there would be w.r.t. any _other_ events that may be driven off of the scheduler tick (i.e. events other than scheduling policies, like timeslice expiration, etc). Perhaps someone else like Thomas, Ingo, or Peter have some input here.
I guess the specific question to ask is: Does the scheduler tick code have any role other than timeslice policies and updating accounting information? Examples would include timer-expiry, for instance. I would think most of this logic is handled by finer grained components like HRT, but I am admittedly ignorant of the actual timer voodoo ;)
Kind Regards,
-Greg
Quoted 23 lines · Gregory Haskins
Hi Rik,
Quoted 8 lines · Rik van Riel
Quoted 2 lines · Gregory Haskins
2) Modify FIFO so that it disables tick by default...update accounting info at next reschedule event.
I like it. The only thing to watch out for is that events that wake up higher-priority FIFO tasks do not get deferred :)
Yeah, agreed. My (potentially half-baked) proposal should work at least from a pure scheduling perspective since FIFO technically does not reschedule based on a tick, and wakeups/migrations should still work bidirectionally with existing scheduler policies.
However, and to what I believe is your point: its not entirely clear to me what impact, if any, there would be w.r.t. any _other_ events that may be driven off of the scheduler tick (i.e. events other than scheduling policies, like timeslice expiration, etc). Perhaps someone else like Thomas, Ingo, or Peter have some input here.
Quoted 6 lines
I guess the specific question to ask is: Does the scheduler tick code have any role other than timeslice policies and updating accounting information? Examples would include timer-expiry, for instance. I would think most of this logic is handled by finer grained components like HRT, but I am admittedly ignorant of the actual timer voodoo ;)
Thinking about this idea some more: I can't see why this isn't just a trivial variation of the nohz idle code already in mainline. In both cases (idle and FIFO tasks) the cpu is "consumed" 100% by some arbitrary job (spinning/HLT for idle, RT thread for FIFO) while we have the scheduler tick disabled. The only real difference is a matter of power-management (HLT/mwait go to sleep-states, whereas spinning/rt-task run full tilt).
Therefore the answer may be as simple as bracketing the FIFO task with tick_nohz_stop_sched_tick() + tick_nohz_restart_sched_tick(). The nohz code will probably need some minor adjustments so it is not assuming things about the state being "idle" (e.g. "isidle") for places when it matters (idle_calls++ stat is one example).
Potential problems:
a) disabling/renabling the tick on a per-RT task schedule() may prove to be prohibitively expensive.
b) we will need to make sure the rt-bandwidth protection mechanism is defeated so the task is allowed to consume 100% bandwidth.
Perhaps these states should be in the cpuset/root-domain, and configured when you create the partition (e.g. "tick=off", "bandwidth=off" makes it an "offline" set).
Kind Regards,
-Greg
Quoted 11 lines · Gregory Haskins
Quoted 11 lines
However, and to what I believe is your point: its not entirely clear to me what impact, if any, there would be w.r.t. any _other_ events that may be driven off of the scheduler tick (i.e. events other than scheduling policies, like timeslice expiration, etc). Perhaps someone else like Thomas, Ingo, or Peter have some input here.
I guess the specific question to ask is: Does the scheduler tick code have any role other than timeslice policies and updating accounting information? Examples would include timer-expiry, for instance. I would think most of this logic is handled by finer grained components like HRT, but I am admittedly ignorant of the actual timer voodoo ;)
There is not much happening in the scheduler tick:
- accounting of CPU time. this can be delegated to some other CPU as long as the user space task is running and consuming 100%
- timer list timers. If there is no service/device active on that CPU then there are no timers to run
- rcu call backs. Same as above, but might need some tweaking.
- printk tick. Not really interesting
- scheduler time slicing. Not necessary in such a context
- posix cpu timers. Only interesting when the application uses them
So there is not much which needs the tick in such a scenario.
Of course we'd need to exclude that CPU from the do_timer duty as well.
Quoted 8 lines
Thinking about this idea some more: I can't see why this isn't just a trivial variation of the nohz idle code already in mainline. In both cases (idle and FIFO tasks) the cpu is "consumed" 100% by some arbitrary job (spinning/HLT for idle, RT thread for FIFO) while we have the scheduler tick disabled. The only real difference is a matter of power-management (HLT/mwait go to sleep-states, whereas spinning/rt-task run full tilt).
Quoted 5 lines
Therefore the answer may be as simple as bracketing the FIFO task with
tick_nohz_stop_sched_tick()+tick_nohz_restart_sched_tick(). The nohz code will probably need some minor adjustments so it is not assuming things about the state being "idle" (e.g. "isidle") for places when it matters (idle_calls++ stat is one example).
Yeah, it's similar to what we do in nohz idle already, but we'd need to split out some of the functions very carefully to reuse them.
Quoted 4 lines
Potential problems:
a) disabling/renabling the tick on a per-RT task
schedule()may prove to be prohibitively expensive.
For a single taks consuming 100% CPU it is a non problem. You disable it once. But yes on a standard system this needs to be investigated.
Quoted 6 lines
b) we will need to make sure the rt-bandwidth protection mechanism is defeated so the task is allowed to consume 100% bandwidth.
Perhaps these states should be in the cpuset/root-domain, and configured when you create the partition (e.g. "tick=off", "bandwidth=off" makes it an "offline" set).
That makes sense and should not be rocket science to implement.
Thanks,
tglx
Quoted 1 line · Thomas Gleixner
That makes sense and should not be rocket science to implement.
I like it and such a thing would do a lot for reducing noise.
However, look at a typical task (from the HPC world) that would be running on an isolated processors. It would
1. Spin on some memory location waiting for an event.
2. Process data passed to it, prepare output data and then go back to 1.
The enticing thing about doing 1 with shared memory and/or infiniband is that it can be done in a few hundred nanoseconds instead of 10-20 microseconds. This allows a much faster IPC communication if we bypass the OS.
For many uses deterministic responses are desired. If the handler that runs is never disturbed by extraneous processing (IPI, faults, irqs etc) then we can say that we run at the maximum speed that the machine can run at. That is what many sites expect.
In an HPC environment synchronization points are essential and the frequency of synchronization points (where we spin on a cacheline) is important for the ability to scale the accuratey and the performance of the algorithm. If we can make N processor operate in a deterministic fashion on f.e. an array of floating point numbers then the rendezvous occurring with minimal wait time in each of the N processes. Getting rid of all sources of interruptions gets us the best performance possible.
Right now often strong variability makes it necessary to have long durations of the processing periods and deal with long wait times because one of the N processes has not finished yet.
Quoted 18 lines · Christoph Lameter
Quoted 1 line · Thomas Gleixner
That makes sense and should not be rocket science to implement.
I like it and such a thing would do a lot for reducing noise.
However, look at a typical task (from the HPC world) that would be running on an isolated processors. It would
1. Spin on some memory location waiting for an event.
2. Process data passed to it, prepare output data and then go back to 1.
The enticing thing about doing 1 with shared memory and/or infiniband is that it can be done in a few hundred nanoseconds instead of 10-20 microseconds. This allows a much faster IPC communication if we bypass the OS.
Quoted 4 lines
For many uses deterministic responses are desired. If the handler that runs is never disturbed by extraneous processing (IPI, faults, irqs etc) then we can say that we run at the maximum speed that the machine can run at. That is what many sites expect.
Right, and I think we can get there. The timer can be eliminated with some work. Faults shouldn't happen on that CPU and all other interrupts can be kept away with proper affinity settings. Softirqs should not happen on such a CPU either as there is neither a hardirq nor a user space task triggering them. Same applies for timers. So there are some remaining issues like IPIs, but I'm pretty sure that they can be tamed to zero as well.
Thanks,
tglx
Quoted 7 lines · Thomas Gleixner
Right, and I think we can get there. The timer can be eliminated with some work. Faults shouldn't happen on that CPU and all other interrupts can be kept away with proper affinity settings. Softirqs should not happen on such a CPU either as there is neither a hardirq nor a user space task triggering them. Same applies for timers. So there are some remaining issues like IPIs, but I'm pretty sure that they can be tamed to zero as well.
There are various timer generated thingies like vm statistics, slab queue management and device specific things that run on each processor.
Quoted 10 lines · Christoph Lameter
Quoted 7 lines · Thomas Gleixner
Right, and I think we can get there. The timer can be eliminated with some work. Faults shouldn't happen on that CPU and all other interrupts can be kept away with proper affinity settings. Softirqs should not happen on such a CPU either as there is neither a hardirq nor a user space task triggering them. Same applies for timers. So there are some remaining issues like IPIs, but I'm pretty sure that they can be tamed to zero as well.
Quoted 2 lines
There are various timer generated thingies like vm statistics, slab queue management and device specific things that run on each processor.
The statistics stuff needs to be tackled anyway as we need to offload the sched accounting to some other cpu.
What slab queue stuff is running on timers and cannot be switched off in such a context?
Device specific stuff should not happen on such a CPU when there is no device handled on it.
Thanks,
tglx
Quoted 2 lines · Thomas Gleixner
The statistics stuff needs to be tackled anyway as we need to offload the sched accounting to some other cpu.
The vm statisticcs in mm/vmstat.c are different from the sched accounting.
Quoted 2 lines
What slab queue stuff is running on timers and cannot be switched off in such a context?
slab does run a timer every 2 second to age queues. If there was activity then there can be a relatively long time in which we periodically throw out portions of the cached data.
Quoted 2 lines
Device specific stuff should not happen on such a CPU when there is no device handled on it.
The device may periodically check for conditions that require action.
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Thomas Gleixner
The statistics stuff needs to be tackled anyway as we need to offload the sched accounting to some other cpu.
The vm statisticcs in
mm/vmstat.care different from the sched accounting.
I know, but the problem is basically the same. Delegate the stats to someone else.
Quoted 6 lines
Quoted 2 lines
What slab queue stuff is running on timers and cannot be switched off in such a context?
slab does run a timer every 2 second to age queues. If there was activity then there can be a relatively long time in which we periodically throw out portions of the cached data.
Right, but why does that affect a CPU which is marked "I'm not involved in that game" ?
Quoted 4 lines
Quoted 2 lines
Device specific stuff should not happen on such a CPU when there is no device handled on it.
The device may periodically check for conditions that require action.
Errm. The device is associated to some other CPU, so why would it require action on an isolated one ? Or are you talking about a device which is associated to that isolated CPU ?
Thanks,
tglx
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
slab does run a timer every 2 second to age queues. If there was activity then there can be a relatively long time in which we periodically throw out portions of the cached data.
Right, but why does that affect a CPU which is marked "I'm not involved in that game" ?
Its run unconditionally on every processor. System needs to scan through all slabs and all queues to figure out if there is something to expire.
Quoted 5 lines
Quoted 1 line
The device may periodically check for conditions that require action.
Errm. The device is associated to some other CPU, so why would it require action on an isolated one ? Or are you talking about a device which is associated to that isolated CPU ?
Some devices now have per cpu threads. See f.e. Mellanox IB drivers. Maybe there is a way to restrict that.
Quoted 3 lines
Some devices now have per cpu threads. See f.e. Mellanox IB drivers. Maybe there is a way to restrict that.
AFAIK the Mellanox drivers just create a single-threaded workqueue.
- R.
Quoted 5 lines · Roland Dreier
Quoted 3 lines
Some devices now have per cpu threads. See f.e. Mellanox IB drivers. Maybe there is a way to restrict that.
AFAIK the Mellanox drivers just create a single-threaded workqueue.
Ok then these per cpu irqs are there to support something different? There are per cpu irqs here. Seems to be hardware supported?
62: 13824 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-0 63: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-1 64: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-2 65: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-3 66: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-4 67: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-5 68: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-6 69: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-7 70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-8 71: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-9 72: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-10 73: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-11 74: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-12 75: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-13 76: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-14 77: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-comp-15 78: 3225 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mlx4-async 79: 1832 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-0 80: 5546 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-1 81: 2604 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-2 82: 124 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-3 83: 743126 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4 84: 857 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5 85: 321 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-6 86: 2296 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-7
Quoted 3 lines
Ok then these per cpu irqs are there to support something different? There are per cpu irqs here. Seems to be hardware supported?
Yes, the driver now creates per-cpu IRQs for completions. However if you don't trigger any completion events then you won't get any interrupts. That's different from the workqueues, which are used to poll the hardware for port changes and internal errors (and which are single-threaded and can be put on whatever "system services" CPU you want)
- R.
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Thomas Gleixner
What slab queue stuff is running on timers and cannot be switched off in such a context?
slab does run a timer every 2 second to age queues. If there was activity then there can be a relatively long time in which we periodically throw out portions of the cached data.
OK, you have me fully confused now.
From other HPC people, I know the "no noise in my math application" requirement. But that means the user code that is running on the CPU must not do anything that wakes the kernel. Not even page faults, so they pin the memory at job start.
Anything the user code does that needs kernel statistics or kernel action is "I must fix my user code", or "I accept that the noise is necessary".
So we don't need to offload stats to other CPUs, stats are not needed.
Quoted 4 lines
Quoted 2 lines
Device specific stuff should not happen on such a CPU when there is no device handled on it.
The device may periodically check for conditions that require action.
Again, what is this device and why is it controlled directly by user-space code. Devices should be controlled even in an HPC environment by the kernel. AFAIK HPC wants the kernel to be the bootstrap and data transfer manager running on a small subset of the total CPUs, with the dedicated CPUs running math jobs.
jim
Quoted 4 lines · jim owens
From other HPC people, I know the "no noise in my math application" requirement. But that means the user code that is running on the CPU must not do anything that wakes the kernel. Not even page faults, so they pin the memory at job start.
Right.
Quoted 5 lines
Anything the user code does that needs kernel statistics or kernel action is "I must fix my user code", or "I accept that the noise is necessary".
So we don't need to offload stats to other CPUs, stats are not needed.
Stats updates are performed if needed or not. Same with slab expiration. Thats why its necessary to Offline the cpu.
Quoted 3 lines · Christoph Lameter
Stats updates are performed if needed or not. Same with slab expiration. Thats why its necessary to Offline the cpu.
Or we fix it to not do anything when not needed.
Come-on Christoph this ain't rocket science.
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Christoph Lameter
Stats updates are performed if needed or not. Same with slab expiration. Thats why its necessary to Offline the cpu.
Or we fix it to not do anything when not needed.
Come-on Christoph this ain't rocket science.
Well lets try it then. Not sure that I can be too much help as long as we have issues with UDP weirdness in the network layers. Bugs before feature work.... Sigh.
Quoted 2 lines · Gregory Haskins
I wonder if it makes sense to do something along the lines of the sched-class...
Disabling the tick isn't a big deal from the scheduler's point of view, its all the other accounting crap that happens.
Quoted 6 lines · Thomas Gleixner
That's just the wrong approach. All you need is a way to tell the kernel that CPUx can switch off the scheduler tick when only one thread is running and that very thread is running in user space. Once another thread arrives on that CPU or the single thread enters the kernel for a blocking syscall the scheduler tick has to be restarted.
That's an elegant approach...I like it.
How would you deal with per-cpu kernel threads (softirqs, etc.) or softirq processing while in the kernel? Switching off the timer tick isn't sufficient because the scheduler will be triggered on the way back to userspace in a syscall.
Chris
Quoted 9 lines · Chris Friesen
Quoted 6 lines · Thomas Gleixner
That's just the wrong approach. All you need is a way to tell the kernel that CPUx can switch off the scheduler tick when only one thread is running and that very thread is running in user space. Once another thread arrives on that CPU or the single thread enters the kernel for a blocking syscall the scheduler tick has to be restarted.
Quoted 4 lines
That's an elegant approach...I like it.
How would you deal with per-cpu kernel threads (softirqs, etc.) or softirq processing while in the kernel?
If you have pinned an interrupt to that CPU then you need to process the softirq for it as well. If that's the device your very single user space thread is talking to then you better want that, if you are not interested then simply pin that device irq to some other CPU: no irq -> no softirq.
Quoted 2 lines
Switching off the timer tick isn't sufficient because the scheduler will be triggered on the way back to userspace in a syscall.
If there is just one user space thread why is the NOOP call to the scheduler interesting ? If you go into the kernel you have some overhead anyway, so why would the few instructions to call schedule() and return with the same task (as it is the only runnable) matter ?
Thanks,
tglx
Quoted 9 lines · Thomas Gleixner
Quoted 2 lines · Chris Friesen
How would you deal with per-cpu kernel threads (softirqs, etc.) or softirq processing while in the kernel?
If you have pinned an interrupt to that CPU then you need to process the softirq for it as well. If that's the device your very single user space thread is talking to then you better want that, if you are not interested then simply pin that device irq to some other CPU: no irq -> no softirq.
Ah, okay. For some reason I had thought that the incoming work was queued up globally and might be handled by any softirq.
Chris
Quoted 5 lines · Thomas Gleixner
You are creating a "one special case" operation mode which is not justified in my opinion. Let's look at the problem you want to solve:
Run exactly one thread on a dedicated CPU w/o any disturbance by the scheduler tick.
Thats not the problem I want to solve. There are multiple events that could disturb a process like timers firing, softirqs and hardirqs.
Quoted 2 lines
You can move away anything else than the scheduler tick from a CPU today already w/o a single line of code change.
How do you remove the per processor kernel threads for allocators and other kernel subsystems? What about IPI broadcasts?
Quoted 10 lines · Chris Friesen
Quoted 8 lines · raz ben yehuda
Quoted 5 lines · Chris Friesen
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Quoted 5 lines
Quoted 1 line
Chris. I offer myself to help anyone wishes to apply OFFSCHED.
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
Given that your entire software is T size , and T' is the amount of real time size, what is the relation T'/T ?
If T'/T << 1 then dissect it, and put the T' in OFFSCHED.
My software T's is about 100MB while the real time section is about 60K. They communicate through a simple ioctls.
CPU isolation example: a transmission engine.
In the image bellow, I am presenting 4 streaming engines, over 4 Intels 82598EB 10Gbps. A streaming engine is actually a Xeon E5420 2.5GHz. Each engine has ***full control*** over its own interface. So you can:
1. fully control the processor's usage.
2. know **exactly*** how much each single packet transmission costs. for example, in this case in processor 3 a single packet average transmission is 1974tscs, which is ~700ns.
3. know how many packets fails to transmit right **on time** ( the Lates counter) . and on time in this case means within the 122us jitter.
4. There are 8 cores in this machine. The rest 4 OS cores are ~95% idle. The only resource these cores share is the bus.
State: kstreamer UP. Started at October 05 05:19:51
******************************************************
CPU 3,63% usage,Sessions 1499,6124301 kbps CPU 5,77% usage,Sessions 1499,6123859 kbps CPU 6,78% usage,Sessions 1498,6123709 kbps CPU 7,73% usage,Sessions 1498,6117766 kbps
Summary: Throughput=24.489Gbps Sessions =5994
******************************************************
Streaming Processor 3
Tx Count : Tot=399990164 Good=399990164 Bad=0 ERR=0(LOC=0,FULL=0)
Time : GoodSendTsc( Max 1565895 Avg 1974) Lates=649
Flow Errors : Underflow (0,0) NotResched=0 GenErr=0
Sessions : Cur 1499(RTP=1499,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0
CPU : 63% usage
Queue : Max Size 92 Avg 69 Csc=79
Throughput (bps) : Tot 6271285040 MPEG 5988905440 Throughput (Mbps): Tot Mbps 6271 MPEG Mbps 5988
Throughput : Packets/sec 568855
Streaming Processor 5
Tx Count : Tot=399944597 Good=399944595 Bad=2 ERR=2(LOC=0,FULL=2)
Time : GoodSendTsc( Max 1566052 Avg 2464) Lates=5521
Flow Errors : Underflow (0,0) NotResched=0 GenErr=0
Sessions : Cur 1499(RTP=1499,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0
CPU : 77% usage
Queue : Max Size 95 Avg 69 Csc=79
Throughput (bps) : Tot 6270832416 MPEG 5988473792 Throughput (Mbps): Tot Mbps 6270 MPEG Mbps 5988
Throughput : Packets/sec 568814
Streaming Processor 6
Tx Count : Tot=399898586 Good=399898585 Bad=0 ERR=0(LOC=0,FULL=0)
Time : GoodSendTsc( Max 1661385 Avg 2474) Lates=8064
Flow Errors : Underflow (0,0) NotResched=0 GenErr=0
Sessions : Cur 1498(RTP=1498,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0
CPU : 78% usage
Queue : Max Size 91 Avg 69 Csc=87
Throughput (bps) : Tot 6270678560 MPEG 5988326400 Throughput (Mbps): Tot Mbps 6270 MPEG Mbps 5988
Throughput : Packets/sec 568800
Streaming Processor 7
Tx Count : Tot=399845166 Good=399845100 Bad=66
ERR=66(LOC=0,FULL=66)
Time : GoodSendTsc( Max 2962620 Avg 2377) Lates=42626
Flow Errors : Underflow (0,0) NotResched=0 GenErr=0
Sessions : Cur 1498(RTP=1498,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0
CPU : 73% usage
Queue : Max Size 94 Avg 69 Csc=66
Throughput (bps) : Tot 6264592672 MPEG 5982514944 Throughput (Mbps): Tot Mbps 6264 MPEG Mbps 5982
Throughput : Packets/sec 568248
--------------- Reservation Load Balancer ------------------ eth2 : 5994501 kbps
eth3 : 5994501 kbps
eth4 : 5990502 kbps
eth5 : 5990502 kbps
Quoted 2 lines
Chris
Quoted 13 lines · raz ben yehuda
Quoted 12 lines · Chris Friesen
Quoted 10 lines · raz ben yehuda
Quoted 5 lines · Chris Friesen
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Chris. I offer myself to help anyone wishes to apply OFFSCHED.
Quoted 8 lines
Quoted 3 lines
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
Given that your entire software is T size , and T' is the amount of real time size, what is the relation T'/T ?
If T'/T << 1 then dissect it, and put the T' in OFFSCHED.
My software T's is about 100MB while the real time section is about 60K.
Chris was stating that your offlined cpu cannot run userspace tasks. How is your answer connected to Chris' statement ? Please stop useless marketing. LKML is about technical problems not advertisement.
Quoted 1 line
They communicate through a simple ioctls.
This is totally irrelevant and we all know how communication channels between Linux and whatever hackery (RT-Linux, RTAI, ... OFFLINE) works.
Quoted 7 lines
CPU isolation example: a transmission engine.
In the image bellow, I am presenting 4 streaming engines, over 4 Intels 82598EB 10Gbps. A streaming engine is actually a Xeon E5420 2.5GHz. Each engine has ***full control*** over its own interface. So you can:
1. fully control the processor's usage.
By disabling the OS control over the CPU resource. How innovative.
Quoted 6 lines
2. know **exactly*** how much each single packet transmission costs. for example, in this case in processor 3 a single packet average transmission is 1974tscs, which is ~700ns.
3. know how many packets fails to transmit right **on time** ( the Lates counter) . and on time in this case means within the 122us jitter.
Are those statistics a crucial property of your OFFLINE thing ?
Quoted 2 lines
4. There are 8 cores in this machine. The rest 4 OS cores are ~95% idle. The only resource these cores share is the bus.
That does not change the problem that you cannot run ordinary user space tasks on your offlined CPUs and you are simply hacking round the real problem which I outlined in my previous mail.
Thanks,
tglx
Quoted 27 lines · Thomas Gleixner
Quoted 13 lines · raz ben yehuda
Quoted 12 lines · Chris Friesen
Quoted 10 lines · raz ben yehuda
Quoted 5 lines · Chris Friesen
We gave it as close to a whole cpu as we could using cpu and irq affinity and we used message queues in shared memory to allow another cpu to handle I/O. In our case we still had kernel threads running on the app cpu, but if we'd had a straightforward way to avoid them we would have used it.
Chris. I offer myself to help anyone wishes to apply OFFSCHED.
Quoted 8 lines
Quoted 3 lines
I just went and read the docs. One of the things I noticed is that it says that the offlined cpu cannot run userspace tasks. For our situation that's a showstopper, unfortunately.
Given that your entire software is T size , and T' is the amount of real time size, what is the relation T'/T ?
If T'/T << 1 then dissect it, and put the T' in OFFSCHED.
My software T's is about 100MB while the real time section is about 60K.Chris was stating that your offlined cpu cannot run userspace tasks. How is your answer connected to Chris' statement ? Please stop useless marketing. LKML is about technical problems not advertisement.
Quoted 5 lines
Quoted 1 line
They communicate through a simple ioctls.
This is totally irrelevant and we all know how communication channels between Linux and whatever hackery (RT-Linux, RTAI, ... OFFLINE) works.
Why are you referring to the above projects as hacks ? What is a hack ?
Quoted 9 lines
Quoted 7 lines
CPU isolation example: a transmission engine.
In the image bellow, I am presenting 4 streaming engines, over 4 Intels 82598EB 10Gbps. A streaming engine is actually a Xeon E5420 2.5GHz. Each engine has ***full control*** over its own interface. So you can:
1. fully control the processor's usage.
By disabling the OS control over the CPU resource. How innovative.
I must say when a john doe like me receives this kind of response from names like Thomas Gleixner it aches.
Quoted 8 lines
Quoted 6 lines
2. know **exactly*** how much each single packet transmission costs. for example, in this case in processor 3 a single packet average transmission is 1974tscs, which is ~700ns.
3. know how many packets fails to transmit right **on time** ( the Lates counter) . and on time in this case means within the 122us jitter.
Are those statistics a crucial property of your OFFLINE thing ?
yes. latency is a crucial property.
Quoted 6 lines
Quoted 2 lines
4. There are 8 cores in this machine. The rest 4 OS cores are ~95% idle. The only resource these cores share is the bus.
That does not change the problem that you cannot run ordinary user space tasks on your offlined CPUs and you are simply hacking round the real problem which I outlined in my previous mail.
OFFSCHED is not just about RT. it is about assigning assignments to another resource outside the operating system. very much like GPUs, network processors, and so on, but just with software that is accessible to the OS.
Quoted 3 lines
Thanks,
tglx
Quoted 6 lines · raz ben yehuda
Quoted 3 lines · Thomas Gleixner
This is totally irrelevant and we all know how communication channels between Linux and whatever hackery (RT-Linux, RTAI, ... OFFLINE) works.
Why are you referring to the above projects as hacks ? What is a hack ?
Everything which works around the real problem instead of solving it.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
2. know **exactly*** how much each single packet transmission costs. for example, in this case in processor 3 a single packet average transmission is 1974tscs, which is ~700ns.
3. know how many packets fails to transmit right **on time** ( the Lates counter) . and on time in this case means within the 122us jitter.
Are those statistics a crucial property of your OFFLINE thing ?
yes. latency is a crucial property.
You are not answering my quesiton.
Also reducing latencies is something we want to do in the kernel proper in the first place. We all know that you can reduce the latencies by taking control away from the kernel and running a side show. But that's nothing new. It has been done for decades already and none of these projects has ever improved the kernel itself.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
4. There are 8 cores in this machine. The rest 4 OS cores are ~95% idle. The only resource these cores share is the bus.
That does not change the problem that you cannot run ordinary user space tasks on your offlined CPUs and you are simply hacking round the real problem which I outlined in my previous mail.
OFFSCHED is not just about RT. it is about assigning assignments to another resource outside the operating system. very much like GPUs, network processors, and so on, but just with software that is accessible to the OS.
I was not talking about RT. I was talking about the problem that you cannot run an ordinary user space task on your offlined CPU. That's the main point where the design sucks. Having specialized programming environments which impose tight restrictions on the application programmer for no good reason are horrible.
Also how are GPUs, network processors related to my statements ? Running specialized software on dedicated hardware which is an addon to the base system controlled by the kernel is not new. There are enough real world applications running Linux on the main CPU and some extra code on an add on DSP or whatever. Cell/SPU or the TI ARM/DSP combos are just the most obvious examples which come to my mind. Where is the point of OFFSCHED here ?
In your earlier mails you talked about isolating cores of the base system by taking the control away from the kernel and what a wonderful solution this is because it allows you full control over that core.
We can dedicate a core to special computations today and we can assign resources of any kind to it under the full control of the OS. The only disturbing factor is the scheduler tick.
So you work around the scheduler tick problem by taking the core away from the OS. That does not solve the problem, it simply introduces a complete set of new problems:
- accounting of CPU utilization excludes the offlined core
- resource assignment is restricted to startup of the application
- standard admin tools (top, ps ....) are not working
- unnecessary restrictions for the application programmer:
- no syscalls
- no standard IPC
- ....
- debugging of the code which runs on the offlined core needs separate tools
- performance analysis e.g. with profiling/performance counters cannot use the existing mechanisms
- ....
Thanks,
tglx
Quoted 1 line · raz ben yehuda
yes. latency is a crucial property.
In the case of network packets, wouldn't you get a lower latency by transmitting the packet from the CPU that knows the packet should be transmitted, instead of sending an IPI to another CPU and waiting for that CPU to do the work?
Inter-CPU communication has always been the bottleneck when it comes to SMP performance. Why does adding more inter-CPU communication make your system faster, instead of slower like one would expect?
Signature · 1 line
All rights reversed.
Quoted 10 lines · Rik van Riel
Quoted 1 line · raz ben yehuda
yes. latency is a crucial property.
In the case of network packets, wouldn't you get a lower latency by transmitting the packet from the CPU that knows the packet should be transmitted, instead of sending an IPI to another CPU and waiting for that CPU to do the work?
Quoted 5 lines
Inter-CPU communication has always been the bottleneck when it comes to SMP performance. Why does adding more inter-CPU communication make your system faster, instead of slower like one would expect?
Maybe just me being paranoid, but from the beginning this "use for dedicated IO processor" has scared the crap out of me.
Reminds me of Winmodem... sell cheap hardware by stealing your CPU!
The HPC FIFO user application on the other hand is a reasonable if somewhat edge-case specialized user batch job.
jim
Quoted 9 lines · Rik van Riel
Quoted 1 line · raz ben yehuda
yes. latency is a crucial property.
In the case of network packets, wouldn't you get a lower latency by transmitting the packet from the CPU that knows the packet should be transmitted, instead of sending an IPI to another CPU and waiting for that CPU to do the work?
Hello Rik
If I understand what you are saying, you say that I pass 1.5K packets to a offline CPU ?
If so, then this is not what I do, because you are very right, it does not make any sense.
I do not pass packets to an offline cpu , i pass assignments. an assignment is a buffer with some context of what do with it (like aio) and a buffer is of ~1MB. Also, the offline processor holds the network interface as it own interface. No two offline processors transmit over a single interface.( I modified the bonding driver to work with offline processor for that ). I am aware of network queue per processors, but benchmarks proved this was better.( I do not have these benchmarks now).
Also these engines do not release any sk_buffs to the operating system, these packets are being reused over and over to reduce latency of allocating memory and cache misses.
Also, in some cases I disabled the transmit interrupts and I released packets ( --skb->users was still greater than 0, not really release ) in an offline context.I learned it from the chelsio driver. This way, I reduced more load from the operating system. It proved to be better in large 1Gbps arrays and was able to remove atomic_inc atomic_dec in some variants of the code, atomic operations cost a lot.
in MSI cards I did not find it useful.in the example i showed, i use MSI and system is almost idle.
Also, as I recall , IPI will not pass to an offladed processor. offsced it runs NMI.
Also, I would to express my apologies if any of this correspondence seems to be as I am trying to PR offsched. I am not.
Quoted 5 lines
Inter-CPU communication has always been the bottleneck when it comes to SMP performance. Why does adding more inter-CPU communication make your system faster, instead of slower like one would expect?
Quoted 10 lines · Andrew Morton
Quoted 1 line · On Wed, 26 Aug 2009 16:40:09 -0400 (EDT) Christoph Lameter…
Peter has not given a solution to the problem. Nor have you.
What problem?
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
Quoted 7 lines
What is the problem statement?
My take on these patches: the kernel gives userspace unmediated access to memory resources if it wants that. The kernel gives userspace unmediated access to IO devices if it wants that. But for some reason people freak out at the thought of providing unmediated access to CPU resources.
Claiming all user-available CPU time from user-space is already
possible: use SCHED_FIFO - the only question are remaining latencies in the final 0.01% of CPU time you cannot claim via SCHED_FIFO.
( Btw., this scheduling feature was implemented in Linux well before
raw IO block devices were implemented, so i'm not sure what you mean by 'freaking out'. )
What we are objecting to are these easy isolation side-hacks for the remaining 0.01% that fail to address the real problem: the latencies. Those latencies can hurt not just isolated apps but _non isolated_ (and latency critical) apps too, and what we insist on is getting the proper fixes, not just ugly workarounds that side-step the problem.
( a secondary objection is the extension and extra layering
of something that could be done within existing APIs/ABIs too. We want to minimize the configuration space. )
Quoted 2 lines
Don't get all religious about this. If the change is clean, maintainable and useful then there's no reason to not merge it.
Precisely. This feature as proposed here hinders the correct solution being implemented - and hence hurts long term maintainability and hence is a no-merge right now. [It also weakens the pressure to fix latencies for a much wider set of applications, hence hurts the quality of Linux in the long run. (i.e. is a net step backwards)]
Ingo
Quoted 8 lines · Ingo Molnar
[...]
Quoted 2 lines
Don't get all religious about this. If the change is clean, maintainable and useful then there's no reason to not merge it.
Precisely. This feature as proposed here hinders the correct solution being implemented - and hence hurts long term maintainability and hence is a no-merge right now.
(Does it "hinder" this in any different way than the following, as in possibly reducing "pressure" for it?)
Quoted 3 lines
[It also weakens the pressure to fix latencies for a much wider set of applications, hence hurts the quality of Linux in the long run. (i.e. is a net step backwards)]
How would you differentiate the above sentiment from "perfect is the enemy of the good"?
- FChE
Quoted 13 lines · Andrew Morton
Quoted 1 line · On Wed, 26 Aug 2009 16:40:09 -0400 (EDT) Christoph Lameter…
Peter has not given a solution to the problem. Nor have you.
What problem?
All I've seen is "I want 100% access to a CPU". That's not a problem statement - it's an implementation.
What is the problem statement?
Quoted 8 lines
My take on these patches: the kernel gives userspace unmediated access to memory resources if it wants that. The kernel gives userspace unmediated access to IO devices if it wants that. But for some reason people freak out at the thought of providing unmediated access to CPU resources.
Don't get all religious about this. If the change is clean, maintainable and useful then there's no reason to not merge it.
thank you Mr Morton. thank you !!!
Quoted 7 lines · Christoph Lameter
Quoted 3 lines
to have sharp teeth nor any apparent poison fangs) - i simply concur with the reasons Peter listed that it is a technically inferior solution.
Ok so you are saying that the reduction of OS latencies will make the processor completely available and have no disturbances like OFFLINE scheduling?
I'm saying that your lack of trying to reduce even low-hanging-fruit latency sources that were pointed out to you fundamentally destroys your credibility in claiming that they are unfixable for all practical purposes.
Or, to come up with a car analogy: it's a bit as if at a repair shop you complained that your car has a scratch on its cooler grid that annoys you, and you insisted that it be outfitted with a new diesel engine which needs no cooler grid (throwing away the nice Hemi block it has currently) - and ignored the mechanic's opinion that he loves the Hemi and that to him the scratch looks very much like bird-sh*t and that a proper car wash might do the trick too ;-)
Quoted 1 line
Peter has not given a solution to the problem. Nor have you.
What do you mean by 'has given a solution' - a patch?
Peter mentioned a few things that you can try to reduce the worst-case latency of the timer tick.
Peter also implemented the hr-tick solution (CONFIG_SCHED_HRTICK) - it's mostly upstream but disabled because it had problems - if you are interested in improving this area you can fix and complete it.
That would benefit ordinary Linux users too, not just rare isolation apps.
Ingo
Quoted 4 lines · Ingo Molnar
I'm saying that your lack of trying to reduce even low-hanging-fruit latency sources that were pointed out to you fundamentally destroys your credibility in claiming that they are unfixable for all practical purposes.
I have never claimed that they are unfixable. However, reducing latencies does not remove a disturbance.
Quoted 7 lines
Or, to come up with a car analogy: it's a bit as if at a repair shop you complained that your car has a scratch on its cooler grid that annoys you, and you insisted that it be outfitted with a new diesel engine which needs no cooler grid (throwing away the nice Hemi block it has currently) - and ignored the mechanic's opinion that he loves the Hemi and that to him the scratch looks very much like bird-sh*t and that a proper car wash might do the trick too ;-)
Nope. Its like you want to get rid of your car and the person you talk to tries to convince you to keep it. He claims if you would just wash it and repair it then it will be maybe become almost invisible and you would have reached your goal of not having a car.
Quoted 2 lines
That would benefit ordinary Linux users too, not just rare isolation apps.
We are talking about apps that need isolation here not regular app.
Ingo Hello
First thank you for your interest.
OFFSCHED is a variant of a proprietary software. it is 4 years old.It is stable. and.. well...this thing works .And it is so simple. SO VERY VERY SIMPLE. ONCE YOU GO OFFLINE YOU NEVER LOOK BACK.
OFFSCHED has a full access to many kernel facilities. My software transmits packets, encrypt packets, and reaches network throughput traffic ( 25Gbs), same as pktgen while saturating its 8 SSD disks.
My software take statistics of an offloaded processor usage, and unlike OS processors, since I have a full control of the processor, the usage is growing quite linearly. there are no bursts of CPU usage. it remains stable of X% usage even when I transmit 25Gbps.
OFFSCHED __oldest__ patch was 4 lines. this how it started. 4 lines of patch and My 2.6.18-8.el5 kernel is suddenly a hard real time kernel. Today, I patch this kernel, build only a bzImage, throw this 2MB bzImage on a server running regular centos/redhat distribution, and caboom, I have a real time server in god-know-where. I do not mess with any driver, i do not mess with initrd. I just fix 4 lines. that all.
OFFSCHED is not just for real time. It can monitor the kernel, protect it and do whatever come to mind. please see OFFSCHED-RTOP.pdf.
thank you
raz
Quoted 57 lines · Ingo aMolnar
Quoted 10 lines · Christoph Lameter
Quoted 8 lines · Ingo Molnar
The thing is, you have cut out (and have not replied to) this
Quoted 4 lines · crutial bit of what Peter
Quoted 4 lines
The past year or so you've been whining about the tick latency, and I've seen exactly _0_ patches from you slimming down the work done in there, even though I pointed out some obvious things that could be done.
Quoted 8 lines
Quoted 4 lines
... which pretty much settles the issue as far as i'm concerned. If you were truly interested in a constructive solution to lower latencies in Linux you should have sent patches already for the low hanging fruits Peter pointed out.
The noise latencies were already reduced in years earlier to the mininum (f.e. the work on slab queue cleaning). Certainly more could be done there but that misses the point.
Peter suggested various improvements to the timer tick related latencies _you_ were complaining about earlier this year. Those latencies sure were not addressed 'years earlier'.
If you are unwilling to reduce the very latencies you apparently cared and complained about then you dont have much real standing to complain now.
( If you on the other hand were approaching this issue with
pragmatism and with intellectual honesty, if you were at the end of a string of patches that gradually improved latencies but couldnt get them below a certain threshold, and if scheduler developers couldnt give you any ideas what else to improve, and _then_ suggested some other solution, you might have a point. You are far away from being able to claim that. )Really, it's a straightforward application of Occam's Razor to the scheduler. We go for the simplest solution first, and try to help more people first, before going for some specialist hack.
Quoted 8 lines
The point of the OFFLINE scheduler is to completely eliminate the OS disturbances by getting rid of *all* OS processing on some cpus.
For some reason scheduler developers seem to be threatened by this idea and they go into bizarre lines of arguments to avoid the issue. Its simple and doable and the scheduler will still be there after we do this.
If you meant to include me in that summary categorization, i dont feel 'threatened' by any such patches (why would i? They dont seem to have sharp teeth nor any apparent poison fangs) - i simply concur with the reasons Peter listed that it is a technically inferior solution.
Ingo
Quoted 2 lines · Christoph Lameter
The point of the OFFLINE scheduler is to completely eliminate the OS disturbances by getting rid of *all* OS processing on some cpus.
No, that's not the point of OFFSCHED. It's about offloading kernel functionality to a peer, and as it currently exists after some years of development. kernel functionality only. Raz has already stated that hard RT is not the point.
<quote> (for full context, jump back a bit in this thread)
Quoted 6 lines
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
In the other words, we might see 'firmwares' that take an entire cpu
for
Quoted 1 line
their usage.
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU.
Processors are to become more and more redundant and Linux as an evolutionary system must use it. why not offload raid5 write engine ? why not encrypt in a different processor ?
Also , having so many processors in a single OS means a bug prone system , with endless contention points when two or more OS processors interacts. let's make things simpler.
</quote>
-Mike
Quoted 74 lines
-----Original Message-----
From: [email protected] [mailto:linux-kernel- [email protected]] On Behalf Of raz ben yehuda
Sent: Wednesday, August 26, 2009 10:54 AM
To: Maxim Levitsky
Cc: Christoph Lameter; Peter Zijlstra; Chris Friesen; Mike Galbraith; [email protected]; [email protected]; andrew motron; [email protected]; lkml; [email protected]
Subject: Re: RFC: THE OFFLINE SCHEDULERQuoted 4 lines · Maxim Levitsky
Quoted 3 lines · Christoph Lameter
Quoted 1 line · raz ben yehuda
How will the kernel is going to handle 32 processors machines ?
These
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
numbers are no longer a science-fiction.
The kernel is already running on 4096 processor machines. Dont worry
about
Quoted 4 lines
Quoted 4 lines
that.
Quoted 2 lines
What i am suggesting is merely a different approach of how to handle multiple core systems. instead of thinking in processes, threads and
so
Quoted 1 line
Quoted 1 line
Quoted 1 line
on i am thinking in services. Why not take a processor and define
this
Quoted 1 line
Quoted 1 line
Quoted 1 line
processor to do just firewalling ? encryption ? routing ?
transmission ?
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
video processing... and so on...
I think that is a valuable avenue to explore. What we do so far is treating each processor equally. Dedicating a processor has benefits
in
Quoted 3 lines
Quoted 3 lines
terms of cache hotness and limits OS noise.
Most of the large processor configurations already partition the
system
Quoted 1 line
Quoted 1 line
using cpusets in order to limit the disturbance by OS processing. A
set of
Quoted 1 line
Quoted 1 line
cpus is used for OS activities and system daemons are put into that
set.
Quoted 2 lines
Quoted 2 lines
But what can be done is limited because the OS threads as well as interrupt and timer processing etc cannot currently be moved. The
ideas
Quoted 2 lines
Quoted 2 lines
that you are proposing are particularly usedful for applications that require low latencies and cannot tolerate OS noise easily (Infiniband
MPI
Quoted 25 lines
Quoted 1 line
base jobs f.e.)
My 0.2 cents:
I have always been fascinated by the idea of controlling another cpu from the main CPU.
Usually these cpus are custom, run proprietary software, and have no datasheet on their I/O interfaces.
But, being able to turn an ordinary CPU into something like that seems to be very nice.
For example, It might help with profiling. Think about a program that can run uninterrupted how much it wants.
I might even be better, if the dedicated CPU would use a predefined reserved memory range (I wish there was a way to actually lock it to that range)
On the other hand, I could see this as a jump platform for more proprietary code, something like that: we use linux in out server platform, but out "insert buzzword here" network stack pro+ can handle 100% more load that linux does, and it runs on a dedicated core....
Quoted 7 lines
Quoted 2 lines
In the other words, we might see 'firmwares' that take an entire cpu for their usage.
This is exactly what offsched (sos) is. you got it. SOS was partly inspired by the notion of a GPU.
Processors are to become more and more redundant and Linux as an evolutionary system must use it. why not offload raid5 write engine ? why not encrypt in a different processor ?
RAID/Encryption + GPU. You got it. This is what one of our teams did but by offloading it on a PCIe I/O module and using couple(Protocol+Application core) of cores. One core could run SAS/SATA and other could run your home grown application f/w and/or a linux distro and you could make it do whatever you want. But that was then. Multi-Core systems are now a commodity.
Chetan Loke
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines
Is it the whole concept of isolating one or more cpus from all normal kernel tasks that you don't like, or just this particular implementation?
I ask because I know of at least one project that would have used this capability had it been available. As it stands they have to live with the usual kernel threads running on the cpu that they're trying to dedicate to their app.
Its the simple fact of going around the kernel instead of using the kernel.
Quoted 6 lines
Going around the kernel doesn't benefit anybody, least of all Linux.
So its the concept of running stuff on a CPU outside of Linux that I don't like. I mean, if you want that, go ahead and run RTLinux, RTAI, L4-Linux etc.. lots of special non-Linux hypervisor/exo-kernel like things around for you to run things outside Linux with.
Out of curiosity, what's the problem with it? Why can't the scheduler be taught to bind one user-space thread on a given CPU and make sure no other threads are scheduled on that CPU? I'm not a scheduler expert but that seems like a logical extension to the current cpuset logic and would help the low-latency workload Christoph has described in the past.
Pekka
Op 25-08-09 21:08, Peter Zijlstra schreef:
Quoted 11 lines · Christoph Lameter
Quoted 9 lines · Mike Galbraith
Quoted 6 lines
I asked the questions I did out of pure curiosity, and that curiosity has been satisfied. It's not that I find it useless or whatnot (or that my opinion matters to anyone but me;). I personally find the concept of injecting an RTOS into a general purpose OS with no isolation to be alien. Intriguing, but very very alien.
Well lets work on the isolation piece then. We could run a regular process on the RT cpu and switch back when OS services are needed?
Quoted 8 lines
Christoph, stop being silly, this offline scheduler thing won't happen, full stop.
Its not a maintainable solution, it doesn't integrate with existing kernel infrastructure, and its plain ugly.
If you want something work within Linux, don't build kernels in kernels or other such ugly hacks.
Hello,
For the one interested in such approach, you can have a look at an now unmaintained project that we developed, ARTiS:
http://www2.lifl.fr/west/artis/
It allows several RT tasks to share a "RT" cpu, and if a task tries to "cheat" by calling a kernel function which disables the preemption or the interrupts, it is temporally migrated to another CPU. This is a working approach, with some good low latency results which can be seen in the papers on the website.
See you,
Eric