Skip to contentSkip to conversation

Attempted summary of "RT patch acceptance" thread

DISCUSSION Paul E. McKenney 17 participants 94 messages Jun 8, 2005 – Jun 14, 2005 Network drivers

lore
Share & tools
lore
#0 lore raw

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned that it might be good to summarize the various approaches. The following is an attempt to do just this, with an eye to providing a reasonable framework for future discussion.

Thoughts? Errors? Omissions?

[Quickly donning the asbestos suit with tungsten pinstripes...]

Thanx, Paul

------------------------------------------------------------------------

CONTENTS

A. INTRODUCTION
B. DESIRABLE PROPERTIES
C. LINUX REALTIME APPROACHES
D. SUMMARY

Search for a line beginning with the corresponding capital letter followed by a period to jump to the corresponding section.

A. INTRODUCTION

Common wisdom dictates that realtime operating systems, particularly hard-realtime operating systems, must be designed from ground up; that serious realtime support cannot be simply grafted onto an existing general-purpose operating system. Although this common wisdom was not arrived at lightly, it is often worthwhile to look for important exceptions to this sort of general rule of thumb. Candidate exceptions include:

1. Many realtime applications use a very restricted subset of
the services provided by a general-purpose OS like Linux. Some applications require realtime support only for scheduling user-mode code, for example, an application that directly accesses MMIO registers mapped into its address space. This observation leads to the possibility of providing very limited realtime support.

2. Computer performance and capacity has increased dramatically
over the past few decades, quite literally by multiple orders of magnitude. A small embedded system can easily be much more capable than a mid-70s supercomputer, for example, the vaunted Cray-1, introduced in 1976, ran at 160MFLOPs and sported 8MB of main memory. In today's terms, this would be a modest embedded system -- and just you try running Linux on an 8MB system! This dramatic increase in performance permits some applications that would have required heavy-duty RTOS support in the 70s to run reasonably well on unmodified general-purpose OSes.

There are still limits to the degree of realtime support that one can expect from a general-purpose OS -- there are some extremely demanding applications that can be satisfied only by hand-coded assembly running on bare metal. In fact, there are applications that can be satisfied only by custom hardware implementations.

Nevertheless, it is clear that Linux can support significant realtime requirements, as it is already being used heavily in the realtime arena. But how far should Linux extend its realtime support, and what is the best way to extend Linux in this direction? Can one approach to realtime satisfy all reasonable requirements, or would it be better to support multiple approaches, each with its area of applicability?

The answers to these questions are not yet clear, and have been the subject of much spirited discussion, for example, see the more than 300 messages in the following LKML thread:

	http://lkml.org/lkml/2005/5/23/156
	http://marc.theaimsgroup.com/?l=linux-kernel&m=111689227213061&w=2

This document looks at some strategies that have been proposed for realtime Linux, comparing and contrasting their capabilities. But, to evaluate these strategies, it is first necessary to determine what exactly one might want in a realtime Linux. If you would rather skip straight to the comparing and contrasting, search for "LINUX REALTIME APPROACHES".

B. DESIRABLE PROPERTIES

As usual, there are conflicting desires, at least they conflict given the current state of the art. These desires fall into the following categories:

1. Quality of service
2. Amount of code that must be inspected to assure quality of service
3. API provided
4. Relative complexity of OS and applications
5. Fault isolation: what non-RT failures endanger RT code?
6. What hardware and software configurations are supported?

Each of these categories is expanded upon below, and later used to compare a number of proposed realtime approaches for Linux. The discussion does go for some time, which is not surprising given that it is summarizing many hundreds of email messages. ;-) Search for the corresponding number at the beginning of a line to skip directly to the discussion of a given category.

1. Quality of Service

The traditional view is that the entire operating system is either hard realtime, soft realtime, or non-realtime, but this viewpoint is too coarse grained. Different workloads have different needs, and there is disagreement over the exact definitions of these three categories of realtime. For example, (at least) the following two definitions of "hard realtime" are in use:

a. In absence of hardware failures, software provably meets
the specified deadlines. This is fine and good, but many applications simply do not need this "diamond hard" realtime.

b. Failure to meet the specified deadline results in application
failure. This is OK, but -only- if there is a corresponding required probability of success. Otherwise, one could claim "hard realtime" by simply failing the application every time it tries to do anything, which is clearly not useful.

A better approach is to simply specified the required probability of meeting the specified deadline in absence of hardware failure. A probability of 1.0 is consistent with definition (a). Other applications will be satisfied with a probability such as 0.999999, which might be sufficiently high that the probability of software scheduling failure is "in the noise" compared with the probability of hardware failure. A recent LKML thread called this "metal hard" realtime. Or was it "ruby hard"? ;-)

Of course, one can increase the reliability of hardware through redundancy, but no hardware configuration provides perfect reliability. For example, clusters can increase reliability, so that the probability of failure of the cluster is p^n, where "p" is the probability of a single node failing and "n" is the number of nodes. Note that this expression never reaches a probability of 1, no matter how large "n" is. In addition, this mathematical expression assumes that the failover software is perfectly reliable and perfectly configured. This assumption conflicts sharply with my own experience, in which there has always been a point beyond which adding nodes -decreased- cluster reliability.

The timeframe is also critically important. Any system can provide hard realtime guarantees if the deadline is an infinite amount of time in the future. No computer system that I am aware of at this writing is capable of meeting a 1-picosecond scheduling deadline for any task of non-zero duration, but then neither can dedicated digital hardware. Some applications have definite response-time goals, for example, industrial process-control applications tend to have response-time goals ranging from 100s of microseconds to small numbers of seconds. Other applications can benefit from any improvement in response-time goals -- faster is better, think in terms of Doom players -- but even in these cases there is normally a point of diminishing returns.

The services used by the realtime application also figure in. Given current disk technology, it is not possible to meet a 100-microsecond deadline for a 1MB synchronous write to disk. Not even if you cheat and supply the disk with a battery-backed-up DRAM. However, many realtime applications need only a few of the services that an operating system might provide. This list might include interrupt handling, process scheduling, disk I/O, network I/O, process creation/destruction, VM operations, and so on. Keep in mind that many popular RTOSes provide very little in the way of services! They frequently leave the complex stuff (e.g., web serving) to general-purpose operating systems.

Note that each service can have an associated deadline that it can meet. The interrupt system might be able to meet a 1-microsecond deadline, the real-time process scheduler a 10-microsecond deadline, the disk I/O system a 10-millisecond deadline for moderate-sized I/Os, and so on. The deadline that a service can meet might also depend on the parameters, so that the disk-I/O system would be expected to take longer for larger I/Os.

Furthermore, the probability might vary from service to service or with the parameters to that service. For example, the probability of network I/O completing successfully in minimal time might well be a function of the number of packets transmitted (to account for the probability of packet loss) as well as of packet size (to account for bit-error rate). To make things even more complicated, the probability of meeting the deadline will vary depending on the length of time allowed. Considering the networking example, a very short deadline might not allow the data transmission to complete, even if it proceeds at wire speed. A longer deadline might allow transmission to complete, but only if there are no transmission errors. An even longer deadline might allow time for a limited number of retransmissions, in order to recover from packet loss due to transmission errors. Of course, a deadline infinitely far into the future would allow guaranteed completion, but I for one am not that patient.

Finally, the performance and scalability of both realtime and non-realtime applications running on the system can be important. Given the current state of the art, one must pay a performance penalty for realtime support, but the smaller the penalty, the better.

So, to sum up, here are the components of a quality-of-service metric for realtime OSes:

a. List of services for which realtime response is supported.

b. For each service:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

ii. Allowable deadline, measured from the time that
the request is initiated to the time by which the response must be received.

c. Performance and scalability provided to both realtime and
non-realtime applications.

2. Amount of Code Inspection Required

So you add a new feature to a realtime operating system. How much of the rest of the system must you inspect and understand in order to be able to guarantee that your new feature provides the required level of realtime response? The smaller this amount of code, the easier it is to add new features and fix bugs, and the greater the number of people who will be able to contribute to the project. In addition, the smaller the amount of such code, the smaller the probability that some well-intentioned bug fix will break realtime response.

Each of the following categories of code might need to be inspected:

a. The low-level interrupt-handing code.

b. The realtime process scheduler.

c. Any code that disables interrupts.

d. Any code that disables preemption.

e. Any code that holds a lock, mutex, semaphore, or other resource
that is needed by the code implementing your new feature.

Of course, use of automated tools could make such inspection much more reliable and less onerous, but such tools would need to deal with the very large number of CPU architectures and configuration options that Linux supports. The smaller the amount of code that must be inspected, the less chance there is that such a tool will fall victim to configuration-architecture combinatorial explosion.

Each of Linux realtime approaches uses a different strategy to minimize the amount of code in these categories. These differences are surprisingly important, and will be discussed in more detail when going over the various approaches to Linux realtime.

3. API Provided

I never have learned to -really- like the POSIX API, with the gets() primitive being a particular cause of heartburn, but given the huge amount of software out there that relies on it and the equally huge number of developers who are familiar with it, one should certainly strive to provide it, or at least a sizeable subset of it.

Other popular APIs include the various Java runtime environments, and of course the feared and loathed, but quite ubiquitous, Windows API.

There are a lot of developers and a lot of software out there. The more of these existing developers and software your API supports, the more successful your realtime facility is likely to be.

4. Relative Complexity

How much realtime capability should be added to the operating system? How much of this burden should the applications take on? Is it better to push some of the complexity into a nanokernel, hypervisor, or other software or firmware layer? Let's first look at the tradeoff between OS and application.

For example, although it is certainly possible to program for separate realtime and non-realtime operating-system instances, doing so adds complexity to the application. Complexity is particularly deadly in the hard realtime arena, and can be literally so if human lives are at risk.

Balancing this consideration is the need for simplicity in the operating-system kernel. This balancing act must be carefully considered, taking both the relative complexities and the number of uses into account. Some would argue that it is worthwhile adding 1,000 lines to the OS if that saves 100 lines in each of 1,000 applications. Others would disagree, perhaps citing the greater fault isolation that might be provided by the separation.

But this balance clearly must be struck somewhere between writing the application to bare metal on the one hand (but achieving a perfectly simple zero-size operating system) and bloating the operating system beyond the limits of maintainability on the other hand.

Similar arguments can be made for moving some functionality into a hypervisor or nanokernel layer, though fault isolation also comes into play here.

Many of the most vociferous arguments seem to revolve around this complexity issue.

5. Fault Isolation

Can a programming error in a non-realtime application or in a non-realtime portion of the OS harm a realtime application?

Some applications do not care: in these cases, a failure anywhere causes a user-visible failure, so it is not important to isolate faults. Of course, even in these cases, it may be valuable to isolate faults in order to aid debugging, but, other than that, the fault isolation does not help overall application reliability.

In other cases, the realtime portion of the application is protecting someone's life and limb, but the non-realtime portion is only compiling statistics and reports. In this case, fault isolation can be of the utmost importance.

What sorts of faults need isolating?

o Excessive disabling of interrupts.

o Excessive disabling of preemption.

o Holding a lock, mutex, or semaphore for too long, when that
resource must be acquired by realtime code.

o Memory corruption, either via wild pointers or via wild DMA.

These faults might occur in the main kernel, in a loadable module, or in some debugging tool, such as a kprobe procedure or a kernel-debugger breakpoint script. Though in the latter case, perhaps realtime deadlines should not be guaranteed when actively debugging. After all, straightforward debugging techniques, such as use of kprint(), can cause response-time problems even in non-realtime environments.

6. Hardware and Software Configurations

Is SMP required? If so, how many CPUs? How many tasks? How many disks? How many HBAs?

If all the code in the kernel were O(1), it might not matter, but the Linux kernel has not yet reached this goal. Therefore, some applications may choose to restrict the software or the hardware configuration of the platform in order to meet the realtime deadlines. This approach is consistent with traditional RTOS methodology -- RTOS vendors have been known to restrict the configurations in which they will support hard realtime guarantees.

C. LINUX REALTIME APPROACHES

The following general approaches to Linux realtime have been proposed, along with many variations on each of these themes:

1. non-CONFIG_PREEMPT
2. CONFIG_PREEMPT
3. CONFIG_PREEMPT_RT
4. Nested OS
5. Dual-OS/Dual-Core
6. Migration Between OSes
7. Migration Within OS

Each of these general approaches is discussed in the following sections. Each section ends with a brief (but perhaps controversial) summary of the corresponding approach's strengths and weaknesses. I do not address "strength of community", even though this may well be the decisive factor. After all, the technical comparision will provide sufficient flame-bait.

This document does not present measured comparisons among all of the approaches, despite the fact that such comparisons would be extremely useful. The reason for this, aside from gross laziness, is that it is wise to agree on the metrics beforehand. Therefore, the comparisons in this document are for the most part qualitative. In some cases, they are based on actual measurements, but these measurements were taken by different people on different configurations using different benchmarks. This is a prime area for future improvement.

1. non-CONFIG_PREEMPT

This is the stock kernel, not even using preemption. Why would -anyone- think of using stock 2.6 for a realtime task? Because some realtime applications have very forgiving scheduling deadlines. One project I worked on in the early 1980s had 2-second response-time deadlines. This was quite a challenge, given that it was running on a 4MHz Z80 CPU -- though, to be fair, the Z80 was accompanied by a hardware floating-point processor that was able to compute a 32-bit floating-point multiply in well under a millisecond. Modern hardware running a stock Linux 2.6 kernel would have no problem with this application. Hey, just having 32 address bits rather than only 16 would help a lot!

a. Quality of service: soft realtime, with timeframe of 10s of
milliseconds for most services. Some I/O requests can take longer. Provides full performance and scalability to both realtime and non-realtime applications.

b. Amount of code that must be inspected to assure quality of service
for a new feature: the entire kernel, every little bit of it, since the entire kernel runs with preemption disabled.

c. API provided: POSIX with limited realtime extensions.
Realtime and non-realtime applications can interact using the normal POSIX services.

d. Relative complexity of OS and applications: everything is
stock, and all the normal system calls operate as expected.

e. Fault isolation: none.

f. Hardware and software configurations supported: all of them.
Larger hardware configurations and some device drivers can result in degraded response time.

Strengths: Simplicity and robustness. "Good enough" realtime support
for undemanding realtime applications. Excellent performance and scalability for both realtime and non-realtime applications. Applications and administrators see a single OS instance.

Weaknesses: Poor realtime response, need to inspect the entire kernel
to find issues that degrade realtime response.

2. CONFIG_PREEMPT

The CONFIG_PREEMPT option renders much of the kernel code preemptible, with the exception of spinlock critical sections, RCU read-side critical sections, code with interrupts disabled, code that accesses per-CPU variables, and other code that explicitly disables preemption.

a. Quality of service: soft realtime, with timeframe of 100s of
microseconds for task scheduling and interrupt handling. System services providing I/O, networking, task creation, and VM manipulation can take much longer. A very small performance penalty is exacted, since spinlocks and RCU must suppress preemption.

b. Amount of code that must be inspected to assure quality of service
for a new feature:

i. The low-level interrupt-handing code.

ii. The process scheduler.

iii. Any code that disables interrupts, which includes
all interrupt handlers, both hardware and softirq.

iv. Any code that disables preemption, including spinlock
critical sections, RCU read-side critical sections, code with interrupts disabled, code that accesses per-CPU variables, and other code that explicitly disables preemption.

v. Any code that holds a lock, mutex, semaphore, or other
resource that is needed by the code implementing the new feature.

c. API provided: POSIX with limited realtime extensions.

d. Relative complexity of OS and applications: all the normal system
calls operate as expected, so realtime and non-realtime processes can interact normally.

e. Fault isolation: none.

f. Hardware and software configurations supported: all of them.
Larger hardware configurations and some device drivers can result in degraded response time.

Strengths: Simplicity. Available now, even from distributions.
Provides "good enough" realtime support for a large number of applications. Applications and administrators see a single OS instance.

Weaknesses: Limited testing, so that some robustness issues remain.
Need to inspect large portions of the kernel in order to find issues that degrade realtime response.

3. CONFIG_PREEMPT_RT

The CONFIG_PREEMPT_RT patch by Ingo Molnar introduces additional preemption, allowing most spinlock (now "mutexes") critical sections, RCU read-side critical sections, and interrupt handlers to be preempted. Preemption of spinlock critical sections requires that priority inheritance be added to prevent the "priority inversion" problem where a low-priority task holding a lock is preempted by a medium-priority task, while a high-priority task is blocked waiting on the lock. The CONFIG_PREEMPT_RT patch addresses this via "priority inheritance", where a task waiting on a lock "donates" its priority to the task holding that lock, but only until it releases the lock. In the example above, the low-priority task would run at high priority until it released the lock, preempting the medium-priority task, so that the high-priority task gets the lock in a timely fashion. Priority inheritance has been used in a number of realtime OS environments over the past few decades, so it is a well-tested concept.

One problem with priority inheritance is that it is difficult to implement for reader-writer locks, where a high-priority writer might wish to donate its high priority to a large number of low-priority readers. The CONFIG_PREEMPT_RT patch addresses this by allowing only one task at a time to read-acquire a reader-writer lock, although it is permitted to do so recursively.

Note that a few critical spinlocks remain non-preemptible, using the "raw spinlock" implementation.

a. Quality of service: soft realtime, with timeframe of about 10
microseconds for task scheduling and interrupt-handler entry. System services providing I/O, networking, task creation, and VM manipulation can take much longer. Since spinlocks are replaced by blocking mutexes, some believe that the performance penalty is significant. There is likely to be some performance penalty exacted from RCU, but, with luck, this penalty will be minimal.

b. Amount of code that must be inspected to assure quality of service
by a new feature:

i. The low-level interrupt-handing code.

ii. The process scheduler.

iii. Any code that disables interrupts, but -not- including
interrupt handlers, which now run in process context.

iv. Any code that disables preemption, including raw-spinlock
critical sections, code with interrupts disabled, code that accesses per-CPU variables, and other code that explicitly disables preemption.

v. Any code that holds a lock, mutex, semaphore, or other resource
that is needed by the code implementing the new feature.

c. API provided: POSIX with limited realtime extensions.

d. Relative complexity of OS and applications: all the normal system
calls operate as expected, so realtime and non-realtime processes can interact normally.

e. Fault isolation: none.

f. Hardware and software configurations supported: most of them.
SMP support is a bit rough, and a number of drivers have not yet been upgraded to work properly in the CONFIG_PREEMPT_RT environment. It is likely that larger hardware configurations and some device drivers can result in degraded scheduling latency, but given that normal spinlocks are now preemptible, this effect should be much less of an issue than for CONFIG_PREEMPT.

Strengths: Excellent scheduling latencies, potential for hard
realtime for some services (e.g., user-mode execution) in some configurations. A number of aspects of this approach might be incrementally added to Linux (e.g., priority inheritance for semaphores to prevent semaphore priority inversion). Applications and administrators see a single OS instance.

Weaknesses: Limited testing, so that robustness issues remain.
Large patch to Linux (~18K lines of context diff). Both realtime and non-realtime applications pay performance and scalability penalties for the realtime service.

4. Nested OS

The Linux instance runs as a user process in an enclosing RTOS. Realtime service is provided by the RTOS, and a richer set of non-realtime services is provided by the Linux instance.

a. Quality of service: hard realtime, with timeframe of about 10
microseconds for services provided by the underlying RTOS. More complex services (I/O, task creation, and so on) will likely take longer to execute, which may impose a significant performance and scalability penalty.

b. Amount of code that must be inspected to assure quality of service
by a new feature:

i. All of the RTOS. One would strive to keep the RTOS
quite small, the greater the number of realtime services provided, the larger the RTOS must be.

ii. Any Linux-kernel code that disables interrupts.
Note that in many implementations, the Linux kernel will be prevented from disabling interrupts, since any attempt to disable interrupts will trap into the RTOS.

If the Linux kernel runs in privileged mode, however, all bets are off. In this case, special care must be used to avoid disabling the real hardware interrupts, including such disabling within any kernel modules that might be loaded.

c. API provided: Whatever the RTOS wants to provide, often a
subset of POSIX with realtime extensions.

d. Relative complexity of OS and applications: there are now two
operating systems, both of which must be configured and administered. Applications that contain both realtime and non-realtime components must be explicitly aware of both OS instances, and of their respective APIs.

e. Fault isolation: the following faults may propagate from the
Linux OS to the underlying RTOS, or not, depending on the implementation:

i. Excessive disabling of interrupts, if the Linux instance
is permitted to disable them (hopefully not).

ii. Memory corruption, if the Linux instance is given direct
access to the hardware MMU or to DMA-capable I/O devices.

f. Hardware and software configurations supported: depends on
the implementation, however, there are products with this architecture that support SMP and a reasonable variety of devices. Note that supporting a large variety of devices either requires that this support be present in the RTOS, or that Linux be granted access to the devices. In the latter case, Linux will likely have the ability to DMA over the top of the RTOS.

Strengths: Excellent scheduling latencies. Hard-realtime support for
some services in some configurations. Reasonable fault isolation for some implementations. Well-tested and robust implementations are available.

Weaknesses: Realtime application software must deal with two separate
OS instances and their respective APIs, with explicit
communication. Administrators must deal with two OS instances.
Non-realtime applications are likely to suffer significant performance and scalability penalties. The pair of cores will be more expensive than a single core, though one might use virtualization to emulate the two CPUs.

5. Dual-OS/Dual-Core

Linux and RTOS instances run side-by-side on different CPUs in the same system. The CPUs might be different physical CPUs, different hardware threads in the same CPU, or different virtual CPUs provided by a virtualizing layer, such as Xen. The two instances might or might not share memory, and, if they do share memory, there might or might not be hardware protection to prevent one OS from overwriting the other OS's memory.

a. Quality of service: hard realtime, with timeframe of about 10
microseconds for services provided by the RTOS. More complex services (I/O, task creation, and so on) will likely take longer to execute. Since the Linux instance runs on a separate core, there need not be any performance or scalability penalty for non-realtime tasks.

b. Amount of code that must be inspected to assure quality of service
by a new feature: all of the RTOS, but only the RTOS. One would strive to keep the RTOS quite small, but the greater the number of realtime services provided, the larger the RTOS must be.

c. API provided: Whatever the RTOS wants to provide, often a
subset of POSIX with realtime extensions.

d. Relative complexity of OS and applications: there are now two
operating systems, both of which must be configured and administered. Applications that contain both realtime and non-realtime components must be explicitly aware of both OS instances and APIs, and must also be aware of whatever hardware facility is used to communicate between the realtime and non-realtime CPUs.

e. Fault isolation: the following faults may propagate from the
Linux OS to the underlying RTOS, or not, depending on the implementation:

i. Memory corruption, but only if the Linux instance is given
direct access to the RTOS's memory or to DMA-capable I/O devices that can access the RTOS's memory.

f. Hardware and software configurations supported: depends on
the implementation, however, there are products based on this approach that support SMP and a reasonable variety of devices.

Strengths: Best possible scheduling latencies with the hardest reasonable
realtime -- just as good as bare metal in some implementations. Best possible fault isolation for some implementations. Well-tested and robust implementations are available. Linux can be used as is, so full performance and scalability can be provided to non-realtime tasks.

Weaknesses: Realtime application software must deal with two separate
OS instances, with explicit communication. Administrators must deal with two OS instances. "RTOSes" that provide the best latencies offer the least services -- in extreme cases, the only service is execution of raw code on bare metal.

6. Migration Between OSes

A Linux and RTOS instance run side-by-side in the same system. The two OSes might run on different physical CPUs, different hardware threads in the same CPU, different virtual CPUs provided by a virtualizing layer like Xen, or alternatively, the two OSes might use some sort of interrupt-pipeline scheme (such as Adeos) to share a single CPU.

However, applications see a single unified environment. Applications run on the RTOS, but the RTOS provides Linux-compatible system calls and memory layout. If the application invokes a non-realtime system call, the task is transparently migrated to the Linux OS instance for the duration of that system call. This differs from the other dual-OS approaches, where the applications must be explicitly aware of the different OSes.

At this writing, it appears that the two instances need to share memory, since tasks can migrate from one OS to the other.

a. Quality of service: hard realtime, with timeframe of about 10
microseconds for services provided by the RTOS. More complex services (I/O, task creation, and so on) will likely take longer to execute. It is also possible for tasks to be "trapped" in the Linux instance, for example, if they are sleeping, but have not yet been given a chance to respond to some event that should wake them up. The performance and scalability penalties to non-realtime tasks can be expected to depend on the amount of protection provided for realtime tasks against non-realtime misbehavior -- the greater the protection, the greater the expected penalty. It may be possible to provide hardware support to improve this tradeoff.

b. Amount of code that must be inspected to assure quality of service
by a new feature:

i. All of the RTOS. One would strive to keep the RTOS
quite small, but the greater the number of realtime services provided, the larger the RTOS must be.

ii. Any Linux-kernel code that disables interrupts.
Note that in many implementations, the Linux kernel will be prevented from disabling interrupts, since any attempt to disable interrupts will trap into the RTOS or into the underlying software/firmware layer (e.g., Xen or Adeos).

If the Linux kernel runs in privileged mode, however, all bets are off. In this case, special care must be used to avoid disabling the real hardware interrupts, including such disabling within any kernel modules that might be loaded.

c. API provided: Full POSIX with realtime extensions. Anytime
a task running in the context of the RTOS attempts to execute a non-realtime system call, it is migrated to the Linux instance.

d. Relative complexity of OS and applications: there are now two
operating systems, both of which must be configured and administered. However, applications can be written as if there was only one OS instance that provided the full set of services, some realtime and some not.

e. Fault isolation: the following faults may propagate from the
Linux OS to the underlying RTOS, or not, depending on the implementation:

i Excessive disabling of interrupts, if the Linux OS
is permitted to disable hardware interrupts (hopefully not, though preventing this may require special hardware).

ii. Memory corruption, either due to wild pointer or
via wild DMA.

f. Hardware and software configurations supported: depends on
the implementation, however, it is reasonable to believe that SMP and a reasonable variety of devices could be supported. Note that supporting a large variety of devices either requires that this support be present in the RTOS, or that Linux be granted access to the devices. In the latter case, Linux will likely have the ability to DMA over the RTOS.

Strengths: Excellent scheduling latencies. Hard-realtime support for
some services in some configurations. Applications see a single OS.

Weaknesses: Administrators must deal with two OS instances.
The two OSes will be extremely sensitive to each other's version and patch level, since they access each other's data structures.

7. Migration Within OS

A Linux instance runs on multiple CPUs, either different physical CPUs, different hardware threads in the same CPU, or different virtual CPUs provided by a virtualizing layer such as Xen. Some (but not all!) of the CPUs are designated as realtime CPUs. If a task running on a realtime CPU executes a trap or system call that contains non-deterministic code sequences, the task is migrated to a non-realtime CPU to complete execution of the trap or system call, then migrated back. This prevents any non-realtime execution of a given realtime task from interfering with that of other realtime tasks.

Interrupts can be directed away from realtime CPUs. Such interrupt redirection is supported on a few architectures, and has in fact been used for realtime support since at least the 2.4 kernel.

a. Quality of service: unknown at present. This approach is
expected to provide hard realtime with timeframe of about 10 microseconds for services provided on the realtime CPUs. More complex services (I/O, task creation, and so on) will likely take longer to execute. It is also possible for tasks to be "trapped" on the non-realtime CPUs, for example, if they are sleeping, but have not yet been given a chance to respond to some event that should wake them up. Since a stock non-CONFIG_PREEMPT Linux may be used, there need be no performance or scalability penalty for non-realtime tasks, nor for realtime tasks that execute only realtime operations. There can be a significant migration penalty when realtime tasks frequently execute non-realtime operations.

b. Amount of code that must be inspected to assure quality of service
by a new feature:

i. Any part of the Linux kernel that is permitted to execute
on the realtime CPUs. This would normally be only the realtime portions of the scheduler and the low-level interrupt and trap handling code (the actual interrupts and traps would be migrated, if necessary).

ii. Any critical section of any lock acquired by the portion
of the Linux kernel that is permitted to execute on the realtime CPUs.

c. API provided: Full POSIX with realtime extensions.

d. Relative complexity of OS and applications: There is but
one OS, though it has a bit of added complexity due to the migration capability. Applications see only one OS.

e. Fault isolation: the following faults may propagate from the
non-realtime CPUs to the realtime CPUs:

i Holding a lock, mutex, or semaphore for too long, when
that resource must be acquired by code that is permitted to run on the realtime CPUs.

ii. Memory corruption, either due to wild pointer or
via wild DMA.

f. Hardware and software configurations supported: all configurations,
though single-CPU systems must have some sort of virtualizing facility so that the OS sees at least two virtual CPUs.

Strengths: Excellent scheduling latencies. Hard-realtime support for
some services in some configurations. Applications and administrators see a single OS and API. Full performance and scalability for non-realtime and for pure-realtime tasks.

Weaknesses: Migration overhead. Strictly theoretical, as no implementations
exist. Requires multiple CPUs, either real or virtual.

D. SUMMARY

At this point, it does not appear that any one approach can be all things to all realtime applications. It is therefore too early to pick a winner. Advocates of a given approach are therefore advised to concentrate their energy on implementations of their favorite approach, rather than engaging in flamewars with advocates of other approaches. ;-)

After all, in the end, the approaches that best meet the needs of the user community will win out. In fact, given that the Linux community has come up with no fewer than seven classes of solutions to a problem that is commonly thought to be unsolvable, it seems quite reasonable to expect that yet more classes of solutions will yet appear.

So, which of these approaches can be combined? The first three can be thought of as elaborations on the general preemption theme, and can be combined with each of the remaining four. The nested-OS and dual-OS/dual-core ideas can be combined by having one of the OSes
on one of the cores have another OS nested within it. The
dual-core/dual-OS approach can be combined with either of the migration approaches, simply by having one of the cores implement the migration approach. It should be possible to combine the two migration approaches, though it is not clear that this is useful.

Regardless of whether Linux's direction ends up being a single one of these approaches, a yet-as-unknown approach, or some combination, realtime Linux looks to remain an exciting area.

#1 lore raw
Quoted 1 line · Paul E. McKenney

[Quickly donning the asbestos suit with tungsten pinstripes...]

Haven't started going through it yet ... but I've just abandonned my usual surroundings and quickly ran into my 10-feet-thick led bunker ;)

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#2 lore raw
Quoted 7 lines · Karim Yaghmour
Quoted 1 line · Paul E. McKenney

[Quickly donning the asbestos suit with tungsten pinstripes...]

Haven't started going through it yet ... but I've just abandonned my usual surroundings and quickly ran into my 10-feet-thick led bunker ;)

Goes to show how naive I am -- I was only expecting the usual flamage, not WMDs. ;-)

Thanx, Paul

#3 lore raw
In reply to Paul E. McKenney

Paul,

I've finished reading your summary and I must say that it's excellent. I don't remember ever reading a non-partisan comparison of this level on the issue of real-time and Linux. Thanks for writing _and_ having the guts to post it :)

There is only one issue I would like to further highlight.

Note: None of the following should be in any way controversial, I'm just providing further background.

Quoted 2 lines · Paul E. McKenney

the corresponding approach's strengths and weaknesses. I do not address "strength of community", even though this may well be the decisive factor.

Indeed what you state here is entirely true. While Adeos and RTAI development has been very active for quite a few years now, it must be said that this development has largely gone unnoticed to LKML participants -- as was obvious by the amount of surprise caused by the realization of the existence of key Adeos and RTAI features.

Part of this is historical. 10 years ago, Linux's state was such that those who were interested in doing rt with it realized that it wasn't about to become rt-capable any time soon. Hence, they "went away" and did their own little thing. They had their mailing lists, their own flame-wars, their own conferences, and there was very little common shared with the mainstream LKML community.

In fact, for a very long time, most kernel developers I spoke to about real-time would refer back to a single project, RTLinux. To this day, actually, if you look in the MAINTAINERS file, it still says:

Quoted 6 lines

RTLINUX REALTIME LINUX

P:      Victor Yodaiken
M:      [email protected]
L:      [email protected]
W:      www.rtlinux.org
S:      Maintained

Yet, the days where RTLinux was _the_ real-time Linux extension are long gone and www.rtlinux.org has been a redirect to a .com site for quite some time now -- I've suggested in the past that this entry be replaced by RTAI, but I was told that neither should in fact be in there, which is fair-enough, but nothing came of this suggestion and the entry is still in the maintainers file.

This state of things remained until May 2002 when I picked up on a post by Andrea to point out a "few" problems the RTAI community saw with the RTLinux project. The ensuing thread was remarkably intense -- not for the faint of heart. Here's the root of it if you're ever interested in reading a huge flame-fest:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102227589127072&w=2
While that discussion did serve to put RTAI on the map for some developers, it also highlighted problems with the RTAI project that needed to be solved.

Part of the issues was the patent problem, and that was solved with the introduction of Adeos. However, with this and other problems solved, the RTAI developers went back the way they came
from: to their own separate mailing lists.

In the past few years, though, a new bread of real-time developers have become interested in making Linux fit for real-time applications. Unlike the previous generation, though, these folks have concentrated their efforts on working within the framework already agreed upon by existing kernel developers: the LKML. And in that, they have achieved a level of awareness amongst the kernel crowd that I think RTAI and Adeos have not yet reached.

I've tried to remedy to this situation as best I can, by pointing out what was obvious to me when appropriate. However, it must be said that I haven't been actively involved with either Adeos or RTAI in quite some time. So while I did play a part in the history of both projects, there are others that are in a much better position than I am to present to the LKML the work done by the RTAI and Adeos communities.

In essence, therefore, what I have to say is this:
- To those who are actively involved in the development of RTAI and Adeos, now is the time to drop the historical tendency of acting as an entirely separate community and to start sharing your work on the LKML.
- To those who are actively involved in finding solutions to the real-time issues in Linux, do not be fooled by the apparent lack of activity in the Adeos or RTAI projects, they are both very active and warrant consideration.

As you correctly state, "strength of community" is likely a decisive factor. What is important here is not to confuse "apparent" strength of community -- or lack thereof -- with "actual" strength of community -- or lack thereof.

Thanks again for a great piece.

Karim Yaghmour

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#7 lore raw
Quoted 3 lines · Chris Friesen

Paul E. McKenney wrote:

Nice writeup.

Glad you liked it!

Quoted 2 lines

I think you mixed up the "weaknesses" section between "nested OS" and "dual-os/dual-core".

Good catch! I have moved the sentence:

The pair of cores will be more expensive than a single core, though one might use virtualization to emulate the two CPUs.

from "nested OS" to "dual-os/dual-core".

Thanx, Paul

#4 lore raw
In reply to Karim Yaghmour
Quoted 7 lines · Karim Yaghmour

Paul,

I've finished reading your summary and I must say that it's excellent. I don't remember ever reading a non-partisan comparison of this level on the issue of real-time and Linux. Thanks for writing _and_ having the guts to post it :)

Thank you for the kind words! But who needs guts when you have senility? ;-)

Quoted 4 lines

There is only one issue I would like to further highlight.

Note: None of the following should be in any way controversial, I'm just providing further background.

;-) ;-) ;-)

Quoted 75 lines
Quoted 2 lines · Paul E. McKenney

the corresponding approach's strengths and weaknesses. I do not address "strength of community", even though this may well be the decisive factor.

Indeed what you state here is entirely true. While Adeos and RTAI development has been very active for quite a few years now, it must be said that this development has largely gone unnoticed to LKML participants -- as was obvious by the amount of surprise caused by the realization of the existence of key Adeos and RTAI features.

Part of this is historical. 10 years ago, Linux's state was such that those who were interested in doing rt with it realized that it wasn't about to become rt-capable any time soon. Hence, they "went away" and did their own little thing. They had their mailing lists, their own flame-wars, their own conferences, and there was very little common shared with the mainstream LKML community.

In fact, for a very long time, most kernel developers I spoke to about real-time would refer back to a single project, RTLinux. To this day, actually, if you look in the MAINTAINERS file, it still says:

Quoted 6 lines

RTLINUX REALTIME LINUX

P:      Victor Yodaiken
M:      [email protected]
L:      [email protected]
W:      www.rtlinux.org
S:      Maintained

Yet, the days where RTLinux was _the_ real-time Linux extension are long gone and www.rtlinux.org has been a redirect to a .com site for quite some time now -- I've suggested in the past that this entry be replaced by RTAI, but I was told that neither should in fact be in there, which is fair-enough, but nothing came of this suggestion and the entry is still in the maintainers file.

This state of things remained until May 2002 when I picked up on a post by Andrea to point out a "few" problems the RTAI community saw with the RTLinux project. The ensuing thread was remarkably intense -- not for the faint of heart. Here's the root of it if you're ever interested in reading a huge flame-fest:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102227589127072&w=2
While that discussion did serve to put RTAI on the map for some developers, it also highlighted problems with the RTAI project that needed to be solved.

Part of the issues was the patent problem, and that was solved with the introduction of Adeos. However, with this and other problems solved, the RTAI developers went back the way they came
from: to their own separate mailing lists.

In the past few years, though, a new bread of real-time developers have become interested in making Linux fit for real-time applications. Unlike the previous generation, though, these folks have concentrated their efforts on working within the framework already agreed upon by existing kernel developers: the LKML. And in that, they have achieved a level of awareness amongst the kernel crowd that I think RTAI and Adeos have not yet reached.

I've tried to remedy to this situation as best I can, by pointing out what was obvious to me when appropriate. However, it must be said that I haven't been actively involved with either Adeos or RTAI in quite some time. So while I did play a part in the history of both projects, there are others that are in a much better position than I am to present to the LKML the work done by the RTAI and Adeos communities.

In essence, therefore, what I have to say is this:
- To those who are actively involved in the development of RTAI and Adeos, now is the time to drop the historical tendency of acting as an entirely separate community and to start sharing your work on the LKML.
- To those who are actively involved in finding solutions to the real-time issues in Linux, do not be fooled by the apparent lack of activity in the Adeos or RTAI projects, they are both very active and warrant consideration.

Quoted 4 lines

As you correctly state, "strength of community" is likely a decisive factor. What is important here is not to confuse "apparent" strength of community -- or lack thereof -- with "actual" strength of community -- or lack thereof.

All good points! I added a few sentences encouraging realtime folks to participate in LKML discussions.

Thanx, Paul

Quoted 8 lines

Thanks again for a great piece.

Karim Yaghmour

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#5 lore raw
In reply to Karim Yaghmour
Quoted 7 lines · Karim Yaghmour

Paul,

I've finished reading your summary and I must say that it's excellent. I don't remember ever reading a non-partisan comparison of this level on the issue of real-time and Linux. Thanks for writing _and_ having the guts to post it :)

I second that. It's refreshing.

Quoted 10 lines

There is only one issue I would like to further highlight.

Note: None of the following should be in any way controversial, I'm just providing further background.

Quoted 2 lines · Paul E. McKenney

the corresponding approach's strengths and weaknesses. I do not address "strength of community", even though this may well be the decisive factor.

[snip][snip][snip]

Quoted 11 lines

In the past few years, though, a new bread of real-time developers have become interested in making Linux fit for real-time applications. Unlike the previous generation, though, these folks have concentrated their efforts on working within the framework already agreed upon by existing kernel developers: the LKML. And in that, they have achieved a level of awareness amongst the kernel crowd that I think RTAI and Adeos have not yet reached.

I've tried to remedy to this situation as best I can, by pointing out what was obvious to me when appropriate.

Ack, and you should be thanked for this. Especially since in some occasions, RTAI, as a community project, did likely get on your nerves, by not presenting its progress in this area on a more regular basis, if it ever bothered presenting it at all. Well, ok, no excuse.

Btw, taking the risk to piss you off a little bit more :o), I don't resist to announce here, two weeks late, that RTAI/fusion (0.7.5) has been ported over ia64, thanks to the contribution of the HYADES project (http://www.hyades-itea.org/).

Quoted 7 lines

However, it must be
said that I haven't been actively involved with either Adeos or RTAI in quite some time. So while I did play a part in the history of both projects, there are others that are in a much better position than I am to present to the LKML the work done by the RTAI and Adeos communities.

Quoted 5 lines

In essence, therefore, what I have to say is this:
- To those who are actively involved in the development of RTAI and Adeos, now is the time to drop the historical tendency of acting as an entirely separate community and to start sharing your work on the LKML.

Needless to remind you that some of us, in the Adeos/RTAI side of the universe, finally acknowledged some time ago that not participating on a regular basis to the LKML was a mistake, and FWIW, I'm one of these people.

This said, the following might help understanding why it has not been as "natural" as one would have thought, for us to have a regular presence so far, in the discussions around the kernel framework:

1. Adeos is basically an enabler for real-time cores that are
steadfastly designed to run side-by-side with the Linux kernel, for
various reasons involving predictability, complexity, scalability and
performances; I won't rehash those here. Meanwhile, no consensus seems to exist among the kernel contributors and users, on the best way to
provide real-time support, either natively inside the kernel, or
cooperatively with, but outside of it, which is obviously a crucial
design issue with long-term implications. People don't even seem to
settle on the definition of "real-time", which indeed covers various application requirements in terms of predictability. Accordingly, /me, as an Adeos contributor, did not see the point in waving hands too much or submit patches for inclusion. Some may say that it's precisely because of the latter that Adeos gets no opportunity of ever being considered by the kernel people, but hey, nobody said I was a smart guy neither.

2. RTAI is a common client of the Adeos layer, and as such, it's an external piece of code which in turn is loaded by the Linux kernel, and runs embodied in plain kernel modules. For the most part, people in the RTAI project usually don't write core kernel code, but are using a limited subset of its documented interface "as is". For this reason, these people tend to keep the noise / signal ratio as low as possible, and do not appear that much on the LKML, just because their contribution might be of no direct interest to the "vanilla" kernel folks.

But,... fact is that RTAI is evolving, and that people working on the
development track called "fusion" have a year ago decided to reach a
seamless cooperation between the Linux kernel and the real-time core for
their users, so that people using RTAI are now able to keep the best
of both worlds. This approach preserves the common Linux programming model in user-space, and offers a consistent integration of the regular Linux services into the RTAI space. Hopefully, this cooperation of real-time designs will someday translate into the cooperation of people promoting these designs.

Accordingly, we already closely track what's going on the LKML on a daily basis, don't doubt of it. All in all, being more present in the kernel community for us is more about finding the best way to submit the most useful information to the LKML, and perhaps get over some shyness to do it. Anyway, we will learn.

Quoted 13 lines

- To those who are actively involved in finding solutions to the real-time issues in Linux, do not be fooled by the apparent lack of activity in the Adeos or RTAI projects, they are both very active and warrant consideration.

As you correctly state, "strength of community" is likely a decisive factor. What is important here is not to confuse "apparent" strength of community -- or lack thereof -- with "actual" strength of community -- or lack thereof.

Thanks again for a great piece.

Karim Yaghmour

Signature · 1 line
Philippe.
#14 lore raw
In reply to Paul E. McKenney
Quoted 8 lines · Paul E. McKenney

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned that it might be good to summarize the various approaches. The following is an attempt to do just this, with an eye to providing a reasonable framework for future discussion.

Thoughts? Errors? Omissions?

I haven't finished it, but it looks great so far. Are you planning to repost it to LKML, or otherwise publish it somewhere, after incorporating feedback?

If you don't mind, I would like to put a copy on the CELF wiki. But if it's going to have a home elsewhere, I don't want to create a duplicate copy.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

#15 lore raw
Quoted 8 lines · Tim Bird
Quoted 7 lines · Paul E. McKenney

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned that it might be good to summarize the various approaches. The following is an attempt to do just this, with an eye to providing a reasonable framework for future discussion.

Quoted 5 lines
Quoted 1 line

Thoughts? Errors? Omissions?

I haven't finished it, but it looks great so far. Are you planning to repost it to LKML, or otherwise publish it somewhere, after incorporating feedback?

Glad you like it!

I figured on sending out an update sometime next week, after incorporating feedback.

Quoted 3 lines

If you don't mind, I would like to put a copy on the CELF wiki. But if it's going to have a home elsewhere, I don't want to create a duplicate copy.

As long as questions, comments, and suggestions get pointed back to me, feel free! I may do a more formal writeup in a couple of months, if there is enough added content of interest.

Thanx, Paul

#16 lore raw
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines · Tim Bird
Quoted 8 lines · Paul E. McKenney

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned that it might be good to summarize the various approaches. The following is an attempt to do just this, with an eye to providing a reasonable framework for future discussion.

Thoughts? Errors? Omissions?

Quoted 8 lines
Quoted 3 lines

I haven't finished it, but it looks great so far. Are you planning to repost it to LKML, or otherwise publish it somewhere, after incorporating feedback?

Glad you like it!

I figured on sending out an update sometime next week, after incorporating feedback.

I believe you are too friendly to vanilla + CONFIG_PREEMPT. People are still seeing tens or hundreds of ms bumps.

Does the LTP include an RT latency test yet?

Lee

#17 lore raw
Quoted 16 lines · Lee Revell
Quoted 15 lines · Paul E. McKenney
Quoted 13 lines · Tim Bird
Quoted 8 lines · Paul E. McKenney

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned that it might be good to summarize the various approaches. The following is an attempt to do just this, with an eye to providing a reasonable framework for future discussion.

Thoughts? Errors? Omissions?

I haven't finished it, but it looks great so far. Are you planning to repost it to LKML, or otherwise publish it somewhere, after incorporating feedback?

Quoted 7 lines
Quoted 4 lines

Glad you like it!

I figured on sending out an update sometime next week, after incorporating feedback.

I believe you are too friendly to vanilla + CONFIG_PREEMPT. People are still seeing tens or hundreds of ms bumps.

Good point -- I certainly need to add a disclaimer to the effect that common hardware (such as VGA, last I checked some months ago) can cause such bumps, as can long-running system calls. Here is my updated CONFIG_PREEMPT quality-of-service section:

a. Quality of service: soft realtime, with timeframe of 100s of
microseconds for task scheduling and interrupt handling, but -only- for very carefully restricted hardware configurations that exclude problematic devices and drivers (such as VGA) that can cause latency bumps of tens or even hundreds of milliseconds (-not- microseconds). Furthermore, the software configuration of such systems must be carefully controlled, for example, doing a "kill -1" traverses the entire task list with tasklist_lock held (see kill_something_info()), which might result in disappointing latencies in systems with very large numbers of tasks. System services providing I/O, networking, task creation, and VM manipulation can take much longer. A very small performance penalty is exacted, since spinlocks and RCU must suppress preemption.

Does this help, or are there other CONFIG_PREEMPT latency issues that need to be called out?

Quoted 1 line

Does the LTP include an RT latency test yet?

Not as far as I know. I believe that LTP contains primarily pass-fail rather than performance tests, but regardless of where RT latency tests live, I believe that there needs to be a good home for them.

There are a number of RT latency tests that people run -- any nominations for particularly good ones? Should we be running lmbench at realtime priority with 15 kernel builds in the background, measuring the latency of each lmbench operation, in addition to running the existing scheduler-latency tests? (Sorry, couldn't resist!)

Thanx, Paul

#18 lore raw

Hello Paul,

kudos for your very nice RT documents (as usual ;)

Quoted 18 lines · Paul E. McKenney

Good point -- I certainly need to add a disclaimer to the effect that common hardware (such as VGA, last I checked some months ago) can

a. Quality of service: soft realtime, with timeframe of 100s of
microseconds for task scheduling and interrupt handling, but -only- for very carefully restricted hardware configurations that exclude problematic devices and drivers (such as VGA) that can cause latency bumps of tens or even hundreds of milliseconds (-not- microseconds). Furthermore, the software configuration of such systems must be carefully controlled, for example, doing a "kill -1" traverses the entire task list with tasklist_lock held (see kill_something_info()), which might result in disappointing latencies in systems with very large numbers of tasks. System services providing I/O, networking, task creation, and VM manipulation can take much longer. A very small performance penalty is exacted, since spinlocks and RCU must suppress preemption.

Quoted 2 lines

Does this help, or are there other CONFIG_PREEMPT latency issues that need to be called out?

You don't need to add it to the document, but as a further pratical example of troublesome hardware besides VGA (could be a software issue and not hardware issue though) I'd like to make the example of the irq handler of the uhci usb1.1 controller that takes up to 8msec on a 1ghz atlhon UP system, and there's nothing that PREEMPT can do about it since it's an hard-irq. This latency keeps triggering a few times per second on my firewall for the last few years.

preempt-RT _can_ do something about it but only _if_ people hacks the drivers properly and makes sure to call local_irq_save_nort instead of local_irq_save and other explicit changes like that, things that if missing are noticeable only during measurements with preempt-RT config option enabled (hence the metal-hard classification of preempt-RT and not ruby-hard definition).

See the tg3 updates required to be safe with preempt-RT without breaking hard-RT as a clear example of how preempt-RT is weak:

drivers/net/tg3.c renamed from drivers/net/tg3.c.orig +2 -2
@@ -3229,9 +3229,9 @@ static int tg3_start_xmit(struct sk_buff
         * So we really do need to disable interrupts when taking
         * tx_lock here.
         */
-       local_irq_save(flags);
+       local_irq_save_nort(flags);
        if (!spin_trylock(&tp->tx_lock)) { 
-               local_irq_restore(flags);
+               local_irq_restore_nort(flags);
                return NETDEV_TX_LOCKED; 
        } 
 

There's no apparent reason why all those changes should be required to get hard-RT.

Both RTAI and rtlinux _don't_ require to change all those drivers to get the guarantee that the kernel will get out of the way within a certain nanoseconds deadline interval.

Furthermore with the scheduler, mutex and context switch code into the equation, it gets more and more difficult to calculate with math the max latency that preempt-RT will provide, while it's almost trivial to do that with RTAI/rtlinux given only the nanokernel code runs before the hard-RT code is invoked and there are not many paths to test, so one has to disable the cache and just measure the few possible nanokenrel paths. (as usual when speaking about hard-RT I've robots in mind, and not audio code that will call into the alsa ioctls)

This below is the kind of stuff where I wouldn't even dream to replace a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly underperformant (cause scheduling hard-irq in userland and scheduling instead of spinning isn't going to be cheap in smp) preempt-RT solution:

http://linuxdevices.com/articles/AT7871136191.html

In the above RTAI should have made it as well as rtlinux of course.

#19 lore raw
Quoted 2 lines · Andrea Arcangeli

See the tg3 updates required to be safe with preempt-RT without breaking hard-RT as a clear example of how preempt-RT is weak:

...

Quoted 2 lines

There's no apparent reason why all those changes should be required to get hard-RT.

Some of it is lack of displine with how various drivers do locking in that they overload the meaning of a spin_lock, etc... to also disable preemption and side effects with preempt_count. Making all of this formal is a good thing since it clarifies and un-ambiguates those code paths. It's something that should have been done in the first place and I consider using implicit masking like that to be rather sloppy.

Quoted 3 lines

Both RTAI and rtlinux _don't_ require to change all those drivers to get the guarantee that the kernel will get out of the way within a certain nanoseconds deadline interval.

They obey the same constraints on driver in those nanokernel systems also apply to the preempt patch. There is no difference other than the regular driver layers in that they now need to be audited, duh. All RTOSs need to do this.

Quoted 9 lines

Furthermore with the scheduler, mutex and context switch code into the equation, it gets more and more difficult to calculate with math the max latency that preempt-RT will provide, while it's almost trivial to do that with RTAI/rtlinux given only the nanokernel code runs before the hard-RT code is invoked and there are not many paths to test, so one has to disable the cache and just measure the few possible nanokenrel paths. (as usual when speaking about hard-RT I've robots in mind, and not audio code that will call into the alsa ioctls)

Quoted 4 lines

This below is the kind of stuff where I wouldn't even dream to replace a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly underperformant (cause scheduling hard-irq in userland and scheduling instead of spinning isn't going to be cheap in smp) preempt-RT solution:

LynxOS is a hard real time OS similar to how the preemption patch works in that it's a single kernel system. It's gets those times regularly and is obviously deterministic because of careful coding conventions just like it has always been. Single kernel determinism is not new. It's only new to you.

bill

#20 lore raw
Quoted 6 lines · Bill Huey

Some of it is lack of displine with how various drivers do locking in that they overload the meaning of a spin_lock, etc... to also disable preemption and side effects with preempt_count. Making all of this formal
is a good thing since it clarifies and un-ambiguates those code paths. It's something that should have been done in the first place

Keep in mind there are large subsystems that are already RT safe like ALSA.

Lee

#21 lore raw
In reply to Bill Huey (hui)
Quoted 5 lines · Bill Huey (hui)

LynxOS is a hard real time OS similar to how the preemption patch works in that it's a single kernel system. It's gets those times regularly and is obviously deterministic because of careful coding conventions just like it has always been. Single kernel determinism is not new. It's only new to you.

Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however, such statements are usually counter-productive.

The existence of real-time monolithic kernels is not disputed. Note, though, that LynxOS is controlled by a single corporation and its development model could not be compared in any way to Linux. Because of Linux's development model, what others are attempting to explain (Andrea included, I think), the reality is that the auditing/upgrading you speak of is unlikely to ever become part of the development philosophy.

I don't contest the fact that those promoting PREEMPT_RT intend to conduct this auditing for the drivers that are of interest to their development. That, though, doesn't mean that those who are maintaining existing driver sets want to take part in that.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#74 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli

preempt-RT _can_ do something about it but only _if_ people hacks the drivers properly and makes sure to call local_irq_save_nort instead of local_irq_save and other explicit changes like that, things that if missing are noticeable only during measurements with preempt-RT config option enabled (hence the metal-hard classification of preempt-RT and not ruby-hard definition).

This is no longer an issue. We did an interrupt disable conversion so now there is only a few places that actually disable interrupts. It hasn't been tested fully, but you should be able to used whatever drivers you want and still get the same interrupt latency. However, preempt latency is still and issue for these drivers.

Daniel

#92 lore raw
In reply to Paul E. McKenney
Quoted 5 lines · Paul E. McKenney
Quoted 1 line

Does the LTP include an RT latency test yet?

Not as far as I know. I believe that LTP contains primarily pass-fail rather than performance tests, but regardless of where RT latency tests live, I believe that there needs to be a good home for them.

Maybe I didn't mean LTP, I'm thinking of whatever that popular benchmark suite is that people use to identify performance regressions. OSDL something or other?

The canonical test is "rtc_wakeup", it just sets up a stream of interrupts from the RTC, polls on it and measures the delay. Check the list archives for the URL.

Lee

#75 lore raw
In reply to Andrea Arcangeli
Quoted 3 lines · Andrea Arcangeli

You don't need to add it to the document, but as a further pratical example of troublesome hardware besides VGA (could be a software issue and not hardware issue though)

The VGA problems were due to (X, not kernel!) driver bugs. Recent versions of X are thought to be OK.

It's the exact same issue described in this paper, scroll down to section 4.5.

http://www.cs.utah.edu/~regehr/papers/hotos7/hotos7.html

There's absolutely nothing the kernel or PREEMPT_RT can do about this, AFAICT even RTAI would be affected, because X lets userspace drivers talk directly to the hardware including wedging the PCI bus.

Lee

#76 lore raw
Quoted 2 lines · Lee Revell

AFAICT even RTAI would be affected, because X lets userspace drivers talk directly to the hardware including wedging the PCI bus.

Yes, I made the usb example exactly to show how latency bugs can be longstanding in the kernel too without requiring X or hardware bugs (this usb thing breaks kernel latency for years and yet nobody fixed it simply because it just works fine in practice, I noticed because apparently my PIT or tsc goes out of sync over time, and in turn my system time was going into the future pretty quick with HZ=1000, or I would have never noticed, of course I'm compiling the kernel with HZ=100 on that system to work around it). Those latency issues can showup in random drivers all the time, and this one was of an extreme magnitude in the millisecond range. The smaller the magnitude of the latency impact, the more frequently you should find it in drivers.

#77 lore raw
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Lee Revell

AFAICT even RTAI would be affected, because X lets userspace drivers talk directly to the hardware including wedging the PCI bus.

Yes, I made the usb example exactly to show how latency bugs can be longstanding in the kernel too without requiring X or hardware bugs (this usb thing breaks kernel latency for years and yet nobody fixed it simply because it just works fine in practice, I noticed because

...

That's because it puts much more stress on the Linux kernel than before. It's going to take some time to log, audit and fix these issues, but fixing them after discovery isn't really a big deal as you make it. The hyperbole where drivers screw with latency is finite. This is a very managable problem. Again, this a problem for all RTOS system and their supporting drivers as well since they often come from BSD open source community.

bill

#8 lore raw
In reply to Paul E. McKenney

I've read the whole summary and really appreciated it. It's clear, precise and objective.

Just a small change to "1 - QoS":

Quoted 5 lines

b. For each service:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

I think it should be (by reference to how you define probability at the beginning of the section):
Probability of not missing any deadline due to software

Thanks again for writing this summary,
Eric

#22 lore raw
In reply to Karim Yaghmour
Quoted 2 lines · Karim Yaghmour

Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however, such statements are usually counter-productive.

Some of the comments from various folks are just intolerably paranoid and I'm frustrated by that.

Quoted 3 lines

The existence of real-time monolithic kernels is not disputed. Note, though, that LynxOS is controlled by a single corporation and its development model could not be compared in any way to Linux. Because of Linux's

Well, as a dude that hangs with the LynxOS engineering staff in cubicals next to me, I can say that development here is really no different than any other kernel community that I've run into, open or closed. We're all kernel engineers except with maybe a difference in emphasis on what is important at any given time.

Quoted 3 lines

development model, what others are attempting to explain (Andrea included, I think), the reality is that the auditing/upgrading you speak of is unlikely to ever become part of the development philosophy.

But it already has. This RT group doesn't need the whole community to jump on it. We can do it ourselves and submit the changes to an upstream trees where relevant.

Quoted 2 lines

I don't contest the fact that those promoting PREEMPT_RT intend to conduct this auditing for the drivers that are of interest to their development.

Yeah, but that's just you and folks that have been friendly to this effort. The rest is another story altogether.

Quoted 2 lines

That, though, doesn't mean that those who are maintaining existing driver sets want to take part in that.

bill

#23 lore raw
Quoted 6 lines · Bill Huey
Quoted 2 lines · Karim Yaghmour

Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however, such statements are usually counter-productive.

Some of the comments from various folks are just intolerably paranoid and I'm frustrated by that.

Oh god,

	http://linuxdevices.com/articles/AT4721170330.html

The original thread and the summary got posted to "linuxdevices" today.

bill

#78 lore raw
In reply to Bill Huey (hui)
Quoted 3 lines · Bill Huey

managable problem. Again, this a problem for all RTOS system and their supporting drivers as well since they often come from BSD open source community.

Did you read Karim's answer at all?

Those RTOS don't have frenetic development in scheduler and all other subsystems and they may be simpler as well, so perhaps you can disable L2 and L3 and measure all possible paths that kernel can take in invoking your rt code, but I doubt you're going to keep it up with linux development in demonstrating the worst case deadline in every possible hardware. Measuring all possible paths of a nanokernel is absolutely trivial in comparison cause there are no different paths, so it's very easy to give a worst case deadline to the RTAI/rtlinux designs, while it sounds a total pain for preempt-RT, even after the local_irq_disable issue is fixed (which is feasible as Daniel seems to have implemented already).

metal hard solutions are fine where having an exact deadline isn't required, or in places where missing the deadline results in a crashed cellphone (even if I really hate it if my cellphone crashes, and yes even the linux cellphone crashes by doing a cp /dev/sda /dev/null, the usbstorage thingy has some problem in accessing the partition table of the internal flash), but again if they would have picked a RTOS for the project I linked on linuxdevices.com that has to react in 50usec, I think they would have done a mistake when ruby-hard solutions like RTAI/rtlinux are available on linux. So kudos to them for their IMHO best correct choice of RT design.

RTOS may be fine for cellphones or other stuff where you may not even need hard-RT at all, but you just want the lowest possible latency, but I'd _never_ use it whenever I need a true deadline, since other more reliable solutions are possible without much more complexity.

But this is me and my focus is on projects like the one quoted on linuxdevices, cellphones may not even need hard-RT at all since I doubt anybody would offload the gsm compression to the main linux cpu with the risk of doing a buffer overflow during decompression and crashing stations. You're free to use a metal-hard solution for problems that requires ruby-hard, but I think that's very wrong choice after you're aware it's not possible to math demonstrate a worst case latency with metal-hard solutions.

About the fact the kernel can crash, and a driver can corrupt memory, that's the difference between ruby and diamond.

I simply think RTOS is an awful solution to hard-RT, RTAI/rtlinux designs are much more reliable. There are certainly areas where RTOS is acceptable, and this is where hard-RT isn't required and you simply want the lowest possible latency when invoking syscalls like alsa ioctls. But when alsa kernel code is out of the equation I'd never use RTOS to get the kernel out of the way.

#79 lore raw
Quoted 2 lines · Andrea Arcangeli

issue is fixed (which is feasible as Daniel seems to have implemented already).

It's in the RT patch as of 0.7.48 . Released early this week .

Daniel

#24 lore raw
In reply to Bill Huey (hui)
Quoted 1 line · Bill Huey

Some of the comments from various folks are just intolerably paranoid

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. How can you tell that? Did you exercise all possible scheduler paths with cache disabled and calculated the worst case latency with rdtsc + math?

Why do you take risks when you can go with much more relaible solutions like RTAI and rtlinux?

Using metal-hard RTOS in things that requires ruby-hard deadline guarantee, seems just an unnecessary risk. I'd prefer you to be more paranoid and less relaxed about these issues, there are enough bugs and risks even when one does the best and is very paranoid.

Said that there are many problems where metal-hard solutions are fine, every single app that don't require a worst-case guarantee of the worst-case latency is fine with metal-hard, and every time one has to call into the alsa ioctl or other complex syscalls also is fine with metal hard since the first kmalloc or lock contention inside RT context, will send your RTOS into water-hard state.

#80 lore raw
In reply to Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

Did you read Karim's answer at all?

Reverse to youe did you read message from the original thread at all ? Karim supports this work and knows this is possible.

Quoted 1 line

Those RTOS don't have frenetic development in scheduler and all other

SCHED_FIFO isn't effected by it, nor is SCHED_RR. That's another hysterical claim on your part.

Quoted 5 lines

subsystems and they may be simpler as well, so perhaps you can disable L2 and L3 and measure all possible paths that kernel can take in invoking your rt code, but I doubt you're going to keep it up with linux development in demonstrating the worst case deadline in every possible hardware. Measuring all possible paths of a nanokernel is absolutely

Not an issue. Folks do it all of the time here. There's no hysteria regarding this here. Single image kernel do it all of the time such as Mach, QNX, LynxOS and friends.

Quoted 4 lines

RTOS may be fine for cellphones or other stuff where you may not even need hard-RT at all, but you just want the lowest possible latency, but I'd _never_ use it whenever I need a true deadline, since other more reliable solutions are possible without much more complexity.

Read the original thread. Folks have intention of doing more than you realize at this point. Trivial strawman claims of provable verses measured latencies are just stupid. The problem is not as wacked you make it.

[cellphone commentary stripped]

Quoted 2 lines

About the fact the kernel can crash, and a driver can corrupt memory, that's the difference between ruby and diamond.

Nanokernels crash as well from lack of memory protection. So what ?

You make too many assumptions in domain you have little experience in and turn it into some assumed fatal argument about single image RT which is far from reality and the true of the matter.

Quoted 2 lines

I simply think RTOS is an awful solution to hard-RT, RTAI/rtlinux designs are much more reliable. There are certainly areas where RTOS is

LynxOS runs on HP printers for real time guarantees all of the time and is good for general purpose usages as well. Folks use single image kernel for RT guarantees even before the existence RTAI and RT Linux. Your history is reversed here. People have been doing this for ages under a single image.

Quoted 4 lines

acceptable, and this is where hard-RT isn't required and you simply want the lowest possible latency when invoking syscalls like alsa ioctls. But when alsa kernel code is out of the equation I'd never use RTOS to get the kernel out of the way.

Hard RT is required in audio. It's require in processing DSP data coming from those cards. ioctl() paths are short and pretty much directly go to the driver.

bill

#25 lore raw

Flames go here (was Re: Attempted summary of "RT patch acceptance" thread)

In reply to Andrea Arcangeli
Quoted 10 lines · Andrea Arcangeli
Quoted 1 line · Bill Huey

Some of the comments from various folks are just intolerably paranoid

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. How can you tell that? Did you exercise all possible scheduler paths with cache disabled and calculated the worst case latency with rdtsc + math?

Why do you take risks when you can go with much more relaible solutions like RTAI and rtlinux?

This thread is converging back to the original "RT patch acceptance" thread. I've changed the subject line so people can distinguish the content from the flames.

Lee

#26 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Andrea Arcangeli

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. How can you tell that? Did you exercise all possible scheduler paths with cache disabled and calculated the worst case latency with rdtsc + math?

Ask Ingo. I'm through with this track and your misinformed comments about it. We just went through a 300+ plus thread discussing the issues and you still apparently haven't read through the relevant portions of it.

Quoted 2 lines

Why do you take risks when you can go with much more relaible solutions like RTAI and rtlinux?

How are they more reliable ? explain that to me ?

Quoted 6 lines

Said that there are many problems where metal-hard solutions are fine, every single app that don't require a worst-case guarantee of the worst-case latency is fine with metal-hard, and every time one has to call into the alsa ioctl or other complex syscalls also is fine with metal hard since the first kmalloc or lock contention inside RT context, will send your RTOS into water-hard state.

When you demonstrate a basic understanding of real time programming and ask a relevant and informed question, then I'll be very happy to explain this patch, single kernel RT design issues. Until then, I'm declaring that I'm officially sick of reply to FUD generating comments from folks that don't understand how this patch works.

bill

#9 lore raw
In reply to Eric Piel
Quoted 2 lines · Eric Piel

I've read the whole summary and really appreciated it. It's clear, precise and objective.

Glad you like it!

Quoted 11 lines

Just a small change to "1 - QoS":

Quoted 5 lines

b. For each service:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

I think it should be (by reference to how you define probability at the beginning of the section):
Probability of not missing any deadline due to software

Good catch! How about the following?

i. Probability of meeting a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Changing "missing" in the original to "meeting".

Thanx, Paul

Quoted 3 lines

Thanks again for writing this summary,
Eric

#81 lore raw
In reply to Bill Huey (hui)
Quoted 4 lines · Bill Huey

LynxOS runs on HP printers for real time guarantees all of the time and is good for general purpose usages as well. Folks use single image kernel for RT guarantees even before the existence RTAI and RT Linux. Your history is reversed here. People have been doing this for ages under a single image.

I don't care what folks did for ages, I care what I think is best to do _now_ with linux, and I don't see why one should go with RTOS when much simpler and more reliable and _more_performant_ solutions exists when switching to linux, even for the printers.

Quoted 2 lines

Hard RT is required in audio. It's require in processing DSP data coming from those cards. ioctl() paths are short and pretty much directly go to

Those are sort of problems where metal-hard may be simpler to deal with, if there's a skip they'll record a second time, and you can just use measurement and proabability there, without providing any guarantee.

Perhaps you can't handle valid criticism cause you have a conflict of interest.

#82 lore raw
Quoted 4 lines · Andrea Arcangeli

I don't care what folks did for ages, I care what I think is best to do _now_ with linux, and I don't see why one should go with RTOS when much simpler and more reliable and _more_performant_ solutions exists when switching to linux, even for the printers.

Covered in the previous mega-flame-war RT acceptance thread.

Quoted 6 lines

Those are sort of problems where metal-hard may be simpler to deal with, if there's a skip they'll record a second time, and you can just use measurement and proabability there, without providing any guarantee.

Perhaps you can't handle valid criticism cause you have a conflict of interest.

Look, I don't like being pissy with you, but this has already been covered in that monster thread from hell. Really, it was already covered.

bill

#93 lore raw
In reply to Lee Revell
Quoted 7 lines · Lee Revell
Quoted 5 lines · Paul E. McKenney
Quoted 1 line

Does the LTP include an RT latency test yet?

Not as far as I know. I believe that LTP contains primarily pass-fail rather than performance tests, but regardless of where RT latency tests live, I believe that there needs to be a good home for them.

Quoted 7 lines

Maybe I didn't mean LTP, I'm thinking of whatever that popular benchmark suite is that people use to identify performance regressions. OSDL something or other?

The canonical test is "rtc_wakeup", it just sets up a stream of interrupts from the RTC, polls on it and measures the delay. Check the list archives for the URL.

OK, http://affenbande.org/~tapas/rtc_wakeup/ is the URL, right?

And LTP might well be a good home for it, I honestly don't know.

I am checking to see if this can be added to the set of tests run in Martin's automated testing. If you haven't looked at this, check out:

http://www.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

This doesn't cover all the various RT variants of Linux, but might focus some attention on mainline latency issues.

Thanx, Paul

#83 lore raw
In reply to Lee Revell
Quoted 8 lines · Lee Revell
Quoted 3 lines · Andrea Arcangeli

You don't need to add it to the document, but as a further pratical example of troublesome hardware besides VGA (could be a software issue and not hardware issue though)

The VGA problems were due to (X, not kernel!) driver bugs. Recent versions of X are thought to be OK.

Quoted 8 lines

It's the exact same issue described in this paper, scroll down to section 4.5.

http://www.cs.utah.edu/~regehr/papers/hotos7/hotos7.html

There's absolutely nothing the kernel or PREEMPT_RT can do about this, AFAICT even RTAI would be affected, because X lets userspace drivers talk directly to the hardware including wedging the PCI bus.

Cute!

I suppose that Eric Piel's approach would avoid this problem if implemented on a NUMA machine with all processes making use of video being affinitied to the NUMA node containing the video card. ;-)

Sounds like I need to add "antisocial hardware" to the list of things that need to be inspected to validate realtime latencies.

Thanx, Paul

#84 lore raw
Quoted 2 lines · Paul E. McKenney

Sounds like I need to add "antisocial hardware" to the list of things that need to be inspected to validate realtime latencies.

And anti-social memory controllers (cough G5 Macs)

bill

#10 lore raw
In reply to Paul E. McKenney
Quoted 13 lines · Paul E. McKenney wrote/
Quoted 11 lines

Just a small change to "1 - QoS":

Quoted 5 lines

b. For each service:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

I think it should be (by reference to how you define probability at the beginning of the section):
Probability of not missing any deadline due to software

Quoted 7 lines

Good catch! How about the following?

i. Probability of meeting a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Changing "missing" in the original to "meeting".

It sounds strange to me (but english is not my mother tongue), it's like hardware was not so good but software helped to recover the situation and, eventually, the deadline was met ;-)

What about using the way you wrote it at the beginning of the section:
"Probability of missing a deadline only because of a hardware failure"

Eric

#27 lore raw
In reply to Bill Huey (hui)
Quoted 7 lines · Bill Huey
Quoted 4 lines · Andrea Arcangeli

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. How can you tell that? Did you exercise all possible scheduler paths with cache disabled and calculated the worst case latency with rdtsc + math?

Ask Ingo. I'm through with this track and your misinformed comments

You didn't provide an answer yourself, and you fallback on somebody else cause there's no valid answer to this question. All I've seen so far are measurements backed by some statistical significance, that's very far from the meaning I give to the word _guarantee_.

I fully acknowledge there are some problems that aren't more equally easily solved with full userland code, for those problems keeping things simpler by making the kernel more RT aware has a value.

I fully acknowledge that simulating infinite cpus has a value too in discovering race conditions too ;).

But I personally dislike all things that works by luck, preempt-RT that aims to provide guarantees about worst case latencies is no exception, so you shoudln't be surprised if I'm not a RTOS backer for problems that can be more easily solved with ruby-hard RT designs like RTAI and rtlinux. I don't care how things have worked in the past on non-linux OS. Most of the time even current not-RT linux will work fine if it's idle as it would be on some embedded usages like the printer example.

This even ignoring the fact all those context switch will not be cheap, kernel can execute a lot more hard-irqs than context switches per second, and this is another fact. On a 4ghz cpu it doesn't matter, but on a embedded it can matter, so the more reliable solution is obviously higher performant too. Of the 50usec it takes to such project on linuxdevices to execute probably a 10% of that is wasted in the irq handling itself (ioapic hardware proper stuff).

Anyway I've more fun things to do on than to talk about hard-RT, which I'm doing just with the aim to provide my little contribution in form of IMHO valid criticism that you clearly don't appreciate.

#85 lore raw
In reply to Bill Huey (hui)
Quoted 5 lines · Bill Huey
Quoted 2 lines · Paul E. McKenney

Sounds like I need to add "antisocial hardware" to the list of things that need to be inspected to validate realtime latencies.

And anti-social memory controllers (cough G5 Macs)

What's wrong with the memory controller on the G5?

#86 lore raw
Quoted 1 line · Zwane Mwaikambo

What's wrong with the memory controller on the G5?

DMA starvation and please don't ask where I got this information. :)

bill

#87 lore raw
Quoted 4 lines · Bill Huey
Quoted 1 line · Zwane Mwaikambo

What's wrong with the memory controller on the G5?

DMA starvation

DMA starvation has also been observed with some PC chipsets during PREEMPT_RT development and the cause was traced to the UDMA133 controller. The symptom is latency traces that seem to go in extreme "slow motion". Check the archives for the hardware affected.

Lee

#91 lore raw
In reply to Andrea Arcangeli
Quoted 3 lines · Andrea Arcangeli

Hello Paul,

kudos for your very nice RT documents (as usual ;)

Thank you, glad you liked it!

Quoted 22 lines · Paul E. McKenney
Quoted 18 lines

Good point -- I certainly need to add a disclaimer to the effect that common hardware (such as VGA, last I checked some months ago) can

a. Quality of service: soft realtime, with timeframe of 100s of
microseconds for task scheduling and interrupt handling, but -only- for very carefully restricted hardware configurations that exclude problematic devices and drivers (such as VGA) that can cause latency bumps of tens or even hundreds of milliseconds (-not- microseconds). Furthermore, the software configuration of such systems must be carefully controlled, for example, doing a "kill -1" traverses the entire task list with tasklist_lock held (see kill_something_info()), which might result in disappointing latencies in systems with very large numbers of tasks. System services providing I/O, networking, task creation, and VM manipulation can take much longer. A very small performance penalty is exacted, since spinlocks and RCU must suppress preemption.

Quoted 2 lines

Does this help, or are there other CONFIG_PREEMPT latency issues that need to be called out?

Quoted 7 lines

You don't need to add it to the document, but as a further pratical example of troublesome hardware besides VGA (could be a software issue and not hardware issue though) I'd like to make the example of the irq handler of the uhci usb1.1 controller that takes up to 8msec on a 1ghz atlhon UP system, and there's nothing that PREEMPT can do about it since it's an hard-irq. This latency keeps triggering a few times per second on my firewall for the last few years.

After reading the ensuing thread, I am not sure what example to use! Maybe I should just pick random examples and see if they tend to get fixed? ;-)

Quoted 40 lines

preempt-RT _can_ do something about it but only _if_ people hacks the drivers properly and makes sure to call local_irq_save_nort instead of local_irq_save and other explicit changes like that, things that if missing are noticeable only during measurements with preempt-RT config option enabled (hence the metal-hard classification of preempt-RT and not ruby-hard definition).

See the tg3 updates required to be safe with preempt-RT without breaking hard-RT as a clear example of how preempt-RT is weak:

drivers/net/tg3.c renamed from drivers/net/tg3.c.orig +2 -2
@@ -3229,9 +3229,9 @@ static int tg3_start_xmit(struct sk_buff
         * So we really do need to disable interrupts when taking
         * tx_lock here.
         */
-       local_irq_save(flags);
+       local_irq_save_nort(flags);
        if (!spin_trylock(&tp->tx_lock)) { 
-               local_irq_restore(flags);
+               local_irq_restore_nort(flags);
                return NETDEV_TX_LOCKED; 
        } 
 

There's no apparent reason why all those changes should be required to get hard-RT.

Both RTAI and rtlinux _don't_ require to change all those drivers to get the guarantee that the kernel will get out of the way within a certain nanoseconds deadline interval.

Furthermore with the scheduler, mutex and context switch code into the equation, it gets more and more difficult to calculate with math the max latency that preempt-RT will provide, while it's almost trivial to do that with RTAI/rtlinux given only the nanokernel code runs before the hard-RT code is invoked and there are not many paths to test, so one has to disable the cache and just measure the few possible nanokenrel paths. (as usual when speaking about hard-RT I've robots in mind, and not audio code that will call into the alsa ioctls)

Quoted 8 lines

This below is the kind of stuff where I wouldn't even dream to replace a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly underperformant (cause scheduling hard-irq in userland and scheduling instead of spinning isn't going to be cheap in smp) preempt-RT solution:

http://linuxdevices.com/articles/AT7871136191.html

In the above RTAI should have made it as well as rtlinux of course.

As long as the nested-OS or migration-between-OSes approaches prevents Linux from disabling interrupts, then interrupts and preemption-disabling are not a problem. As noted later in this thread, there is still the possibility of hardware stalls, which seems to affect all the approaches, with the possible exception of migration-within-OS between CPUs that don't share the offending hardware. Maybe a dual AMD?

Thanx, Paul

#11 lore raw
In reply to Eric Piel
Quoted 21 lines · Eric Piel
Quoted 20 lines · Paul E. McKenney wrote/
Quoted 5 lines

Just a small change to "1 - QoS":

Quoted 2 lines

b. For each service:

Quoted 7 lines
Quoted 3 lines

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

I think it should be (by reference to how you define probability at the beginning of the section):
Probability of not missing any deadline due to software

Good catch! How about the following?

i. Probability of meeting a deadline due to software,
ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Quoted 5 lines
Quoted 1 line

Changing "missing" in the original to "meeting".

It sounds strange to me (but english is not my mother tongue), it's like hardware was not so good but software helped to recover the situation and, eventually, the deadline was met ;-)

Hmmm... It would appear that English's being my mother tongue is not helping me as much as one might hope. ;-)

Quoted 2 lines

What about using the way you wrote it at the beginning of the section:
"Probability of missing a deadline only because of a hardware failure"

Good point, I may just need to invert the whole thing, so that it becomes something like:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 0 corresponding to the hardest possible hard realtime.

But then the "p^n" becomes "1-(1-p)^n". Bleah.

OK, how about the following?

i. Probability of meeting a deadline in absence of hardware
failure, ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Thanx, Paul

#88 lore raw
In reply to Bill Huey (hui)
Quoted 5 lines · Bill Huey
Quoted 2 lines · Paul E. McKenney

Sounds like I need to add "antisocial hardware" to the list of things that need to be inspected to validate realtime latencies.

And anti-social memory controllers (cough G5 Macs)

OK, the list now reads:

Each of the following categories of code might need to be inspected:

a. The low-level interrupt-handing code.

b. The realtime process scheduler.

c. Any code that disables interrupts.

d. Any code that disables preemption.

e. Any code that holds a lock, mutex, semaphore, or other
resource that is needed by the code implementing your new feature.

f. Any code that manipulates hardware that can stall the
bus, delay interrupts, or otherwise interfere with forward progress. Note that it is also necessary to inspect user-level code that directly manipulates such hardware.

I added point "f". Does that cover it?

Thanx, Paul

#12 lore raw
In reply to Paul E. McKenney
Quoted 5 lines · Paul E. McKenney wrote/
Quoted 2 lines · Eric Piel

What about using the way you wrote it at the beginning of the section:
"Probability of missing a deadline only because of a hardware failure"

Quoted 8 lines

Good point, I may just need to invert the whole thing, so that it becomes something like:

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 0 corresponding to the hardest possible hard realtime.

But then the "p^n" becomes "1-(1-p)^n". Bleah.

Yes, it seems language doesn't fit well with mathematics ;-)

Quoted 7 lines

OK, how about the following?

i. Probability of meeting a deadline in absence of hardware
failure, ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Sounds good!

Eric

#28 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Bill Huey
Quoted 4 lines · Andrea Arcangeli

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. How can you tell that? Did you exercise all possible scheduler paths with cache disabled and calculated the worst case latency with rdtsc + math?

Quoted 6 lines
Quoted 1 line

Ask Ingo. I'm through with this track and your misinformed comments

You didn't provide an answer yourself, and you fallback on somebody else cause there's no valid answer to this question. All I've seen so far are measurements backed by some statistical significance, that's very far from the meaning I give to the word _guarantee_.

I just -know- I am going to regret stepping into the middle of this one...

[Hey, Karim!!! Any extra room in that bunker of yours?]

All I have seen Ingo claim is that -some- hardware configurations running -some- specific workloads had excellent -measured- maximum scheduling latencies. I have not seen Ingo claim that he has mathematically proven that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let alone make any claim of what such a mathematically proven bound might be. Nor have I seen Ingo claim that CONFIG_PREEMPT_RT has passed any sort of industry-standard acceptance test. If I missed such a mathematical proof, or such an industry-standard acceptance test, please send me the URL.

Ingo -has- greatly reduced the amount of kernel code that must be inspected to guarantee realtime response, and this is a great accomplishment and a great contribution. I believe that his approach is more than "good enough" for a great number of applications that are traditionally thought of as "hard realtime". But, unless I missed something, it is not mathematically proven, nor is it in any way "certified".

Some of the approaches involving separate RTOSes -can- legitimately claim have mathematically proven worst-case scheduling latencies. For example, the dual-OS/dual-core approach can do so, particularly in the case where the "RTOS" is a tight loop written on bare metal in hand-coded assembly. One might consider this to be a trivial example, perhaps even a stupid example, but there are a lot of apps out there that use exactly this approach.

The migration-between-OSes approach, such as RTAI Fusion, might also be able to mathematically guarantee RTOS scheduling latencies. And, unlike the dual-OS/dual-core approaches, it does provide a single environment to applications. However, it still results in two separate OS instances to administer, and, as near as I can tell, the RTOS has to stick its hands so deeply into Linux's internals that it might as well be part of the Linux kernel. Unless I am missing something, any sort of bug in the Linux kernel has a reasonable chance of messing up the RTOS, since the RTOS must paw through Linux's tasks.

Can we say that one of these approaches is definitely "good enough" for all reasonable Linux RT work, and that we should therefore stop working on the other approach?

I might well be missing something, but I don't believe so, at least not yet.

In the meantime, the more time anyone spends sending flames on LKML, the less time that person is putting into improving his/her favorite approach. So the greater number of flames one sends, the smaller one's favorite approach's chance of success!

Thanx, Paul

Quoted 27 lines

I fully acknowledge there are some problems that aren't more equally easily solved with full userland code, for those problems keeping things simpler by making the kernel more RT aware has a value.

I fully acknowledge that simulating infinite cpus has a value too in discovering race conditions too ;).

But I personally dislike all things that works by luck, preempt-RT that aims to provide guarantees about worst case latencies is no exception, so you shoudln't be surprised if I'm not a RTOS backer for problems that can be more easily solved with ruby-hard RT designs like RTAI and rtlinux. I don't care how things have worked in the past on non-linux OS. Most of the time even current not-RT linux will work fine if it's idle as it would be on some embedded usages like the printer example.

This even ignoring the fact all those context switch will not be cheap, kernel can execute a lot more hard-irqs than context switches per second, and this is another fact. On a 4ghz cpu it doesn't matter, but on a embedded it can matter, so the more reliable solution is obviously higher performant too. Of the 50usec it takes to such project on linuxdevices to execute probably a 10% of that is wasted in the irq handling itself (ioapic hardware proper stuff).

Anyway I've more fun things to do on than to talk about hard-RT, which I'm doing just with the aim to provide my little contribution in form of IMHO valid criticism that you clearly don't appreciate.

#13 lore raw
In reply to Eric Piel
Quoted 9 lines · Eric Piel
Quoted 8 lines · Paul E. McKenney wrote/
Quoted 2 lines · Eric Piel

What about using the way you wrote it at the beginning of the section:
"Probability of missing a deadline only because of a hardware failure"

Good point, I may just need to invert the whole thing, so that it becomes something like:

Quoted 6 lines
Quoted 5 lines

i. Probability of missing a deadline due to software,
ranging from 0 to 1, with the value of 0 corresponding to the hardest possible hard realtime.

But then the "p^n" becomes "1-(1-p)^n". Bleah.

Yes, it seems language doesn't fit well with mathematics ;-)

Hey, it could be worse. I am just glad that it is not normally necessary to do integration by parts, trig substitutions, or Laplace transforms on English sentences!!! ;-)

Quoted 7 lines
Quoted 6 lines

OK, how about the following?

i. Probability of meeting a deadline in absence of hardware
failure, ranging from 0 to 1, with the value of 1 corresponding to the hardest possible hard realtime.

Sounds good!

OK, I made the change! Thank you for catching this one, good eyes!

Thanx, Paul

#29 lore raw
In reply to Paul E. McKenney
Quoted 1 line · Paul E. McKenney

[Hey, Karim!!! Any extra room in that bunker of yours?]

If you bring your own doritos and coffee, there just might be some room to sleep beneath the computer-controlled mechanical saw bench I've set up to evaluate the real-time responsiveness of Linux ...

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#30 lore raw
Quoted 7 lines · Karim Yaghmour
Quoted 1 line · Paul E. McKenney

[Hey, Karim!!! Any extra room in that bunker of yours?]

If you bring your own doritos and coffee, there just might be some room to sleep beneath the computer-controlled mechanical saw bench I've set up to evaluate the real-time responsiveness of Linux ...

Cool! I will bring a few packages of hot dogs!!!

Thanx, Paul

#72 lore raw
In reply to Andrea Arcangeli
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line · Bill Huey

Some of the comments from various folks are just intolerably paranoid

Just tell me how can you go to a customer and tell him that your linux-RTOS has a guaranteed worst case latency of 50usec. [...]

Andrea, please just stop these scare tactics, it's getting boring. First you came with incorrect statements about locks, then you came with the patent boogeyman, then you came with the driver bogosity, and now you come with "but ... [sputter] ... but ... where's the _guarantee_?".

Andrea, _please_ lean back and read back some of your earlier arguments. To make it easier for you and to refresh your recollection, let me give you a selection, just so that you can see why your sentiment creates flames and personal responses:

'[...] how can preempt-RT ever become hard-RT when a simple lock hangs
it.' (Andrea Arcangeli, May 31, 2005)

'Exactly, they're simply not remotely comparable, a VM improvement may
break preempt-RT anytime, it's just too easy to screw things up and invalidate all "measurements".' (Andrea Arcangeli, Jun 1, 2005)

'Indeed, that's why I believe hard-RT with preempt-RT is just a
joke.' (Andrea Arcangeli, May 31, 2005)

'Then I'm afraid preempt-RT infringe on the patent that they take
after years of doing that in linux. I'm not a lawyer but you may want to check before investing too much on this for the next 15 years.' (Andrea Arcangeli, Jun 1, 2005)

'Why do you take risks when you can go with much more relaible
solutions like RTAI and rtlinux?' (Andrea Arcangeli, Jun 11, 2005)

I have to say, based on these statements you could soon become a fine replacement for Blake Stowell or the Iraqi Information Minister :-)

just to make it plain and obvious: there are two things that are true about the above snippets: 1) as far as they were technical comments they are all dead wrong, and 2) you never apologized for them. Wouldnt you yourself become a bit ... touchy if this happened to you on such a widespread basis? Please give me an answer, how much unjust accusations do i have to take before i have the _right_ to flame you, hm? :-)

thinking about the root causes of your behavior, it seems to me that you have a real inner trouble admitting mistakes (and that's not only true for this incident) - even though mistakes are human and i do at least a dozen mistakes every day and sometimes i screw up really bad. Like 3 months ago when i criticised SECCOMP [your project] in an uninformed and thus unjust way. Perhaps we are getting this from you because you are still hurting from that SECCOMP incident? If it's about SECCOMP then i'd like to apologize again: i was wrong about SECCOMP and it's a fine and promising project. Okay?

if you review the PREEMPT_RT technology (with a cool mind) i do think you'll eventually come to the conclusion that it is actually a pretty nifty idea and implementation, with some pretty good potential :-) It's not trying to be the holy grail, it wont (nor does it want to) replace nanokernels, but it actually has some thinking behind it and is definitely useful to alot of people. And that is what matters. Your feelings about it or me dont really count.

Ingo

#89 lore raw
In reply to Paul E. McKenney
Quoted 7 lines · Paul E. McKenney

f. Any code that manipulates hardware that can stall the
bus, delay interrupts, or otherwise interfere with forward progress. Note that it is also necessary to inspect user-level code that directly manipulates such hardware.

I added point "f". Does that cover it?

Yes. F is really a bad problem if it really can cause DMA starvation on the memory bus on some arch as stated on this thread. Hopefully measurements are good enough to rule it out and there will be no corner cases triggering once in a while.

#31 lore raw
In reply to Paul E. McKenney
Quoted 2 lines · Paul E. McKenney

I just -know- I am going to regret stepping into the middle of this one...

Never mind, I keep regretting it myself. Perhaps it was better if I avoided posting here, and by now people would think preempt-RT was the holy grail of hard-RT and that would obsolete RTAI/rtlinux, when infact it's an inferior solution from from a reliability/guarantee standpoint and from a performance standpoint too (as some measurements showed already too) [and local_irq_disable would still be not emulated in software]

Quoted 5 lines

All I have seen Ingo claim is that -some- hardware configurations running -some- specific workloads had excellent -measured- maximum scheduling latencies. I have not seen Ingo claim that he has mathematically proven that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let alone make any claim of what such a mathematically proven bound might be.

Indeed. As I stated in my first relevant post to this thread on "Tue, 31 May 2005 18:11:57 +0200" _before_ most of the other quotes that I received today, and before realizing the _irq part of the spin_lock_irq was already emulated in software and before realizing that the rtlinux patent doesn't forbid to emulate in software the _irq part of the spin_lock_irq because I'm not really good at reading all the patent claims but I focused only on the patent concept (and regardless there is apparently somebody who rightfully paid lawyers for research in that area):

For the determinism, you could do what Ingo did so far, that is to "measure" but there's no way a "measurement" can provide an hard-RT guarantee. The "measure" way is great for the lowlatency patches, and to try to eliminate the bad-latencies paths, but it _can't_ guarantee a "worst-case-latency".

This is what I posted in my _second_ email to this thread. Of course nobody is quoting the above, they picked other later emails, but if you go back to the second email (and first relevant one) I posted here, you'll see this text at the end. Sorry for that mistake about the _irq part and the related patent claims that I'm not good at reading cause it's written by lawyers, but that mistake made no difference to the bottom line cause local_irq_disable was still disable the hard irqs without being emulated in software. I was reading local_irq_disable that wasn't touched (local_irq_disable is the _first_ thing I check in every RT patch on earth) and so I assumed no _irq disable was emulated. My bad but it didn't change the bottom line.

Now even local_irq_disable can be emulated in software (incidentally happened after my suggestions last week) but that still leaves the above issue in my second email unsolved.

It's not like I made up this "guarante" of worst case latency argument after realizing that _irq was emulated and after local_irq_disable has been emulated too, as somebody apparently seems to have described today by providing a list of quotes ignoring the above argument that was the core of my first relevant email to this thread.

If you check the timestamp, I mentioned the above core issue, way _before_ going into the _irq details.

Quoted 3 lines

Ingo -has- greatly reduced the amount of kernel code that must be inspected to guarantee realtime response, and this is a great accomplishment and a great contribution. I believe that his approach

Indeed.

Quoted 4 lines

is more than "good enough" for a great number of applications that are traditionally thought of as "hard realtime". But, unless I missed something, it is not mathematically proven, nor is it in any way "certified".

Yes, it could be that 99% of the app that traditionally claims to need hard-RT will be fully covered by preempt-RT (assuming the cpu can keep up with the slowdown), but when I talk about hard-RT I only think at those apps like the linuxdevices article that beats at 50usec. To me hard-RT is all about guaranteeing to provide a deadline, the rest I call it lowlatency patches. preempt-RT is obviously a very great improvement to the lowlatency soft-RT approach and I agree a lot of those apps thought to need hard-RT will be fine with it.

But one should evaluate if an hard-RT guarantee is truly needed or not before picking preempt-RT, the performance should be evaluated as well cause irq handling is much faster than a context switch.

The slowdown numbers posted so far are quite impressive as well.

Think that in some recent research I'm doing, I'm looking to make the semaphore a spinlock to boost performance, preempt-RT is the opposite ;)

Quoted 8 lines

Some of the approaches involving separate RTOSes -can- legitimately claim have mathematically proven worst-case scheduling latencies. For example, the dual-OS/dual-core approach can do so, particularly in the case where the "RTOS" is a tight loop written on bare metal in hand-coded assembly. One might consider this to be a trivial example, perhaps even a stupid example, but there are a lot of apps out there that use exactly this approach.

Quoted 9 lines

The migration-between-OSes approach, such as RTAI Fusion, might also be able to mathematically guarantee RTOS scheduling latencies. And, unlike the dual-OS/dual-core approaches, it does provide a single environment to applications. However, it still results in two separate OS instances to administer, and, as near as I can tell, the RTOS has to stick its hands so deeply into Linux's internals that it might as well be part of the Linux kernel. Unless I am missing something, any sort of bug in the Linux kernel has a reasonable chance of messing up the RTOS, since the RTOS must paw through Linux's tasks.

Yes. I'm a beliver in the dual-OS/dual-core model, as the only one who can reasonably simply guarantee a deadline.

The single image RTOS has much larger pieces of complex code involved in the RT equation (in your list you didn't mention the mutex code itself), plus this codes like the scheduler changes all the time (it's not like the irq code that pratically never changes in stable cycles), and it slowsdown quite a bit as well at emulating infinite cpus and overscheduling at every contention.

Quoted 6 lines

Can we say that one of these approaches is definitely "good enough" for all reasonable Linux RT work, and that we should therefore stop working on the other approach?

I might well be missing something, but I don't believe so, at least not yet.

I agree with you. Perhaps I didn't make my point of view clear enough on the good things of preempt-RT. preempt-RT is a great feature for some apps.

I _only_ criticise people who wants to put preempt-RT on par with guaranteed hard-RT solutions like RTAI/rtlinux that can easily provide a guaranteed deadline, while preempt-RT can only proviede measurements backed by statistically signficance, which is quite far away from a guarantee. Plus it's much slower as well. But apparently some people here, wants you to believe that preempt-RT will be exactly guaranteed hard-RT as RTAI/rtlinux, and they can't handle it when I say this is not the case. This is the only thing I'm objecting as stated in my first relevant email of the thread partly quoted above.

I'm sure I've ever said that preempt-RT is useless and there's no application for it, infact I stated _multiple_ times there are cases like alsa where preempt-RT is the very best that one can do.

Quoted 2 lines

In the meantime, the more time anyone spends sending flames on LKML, the less time that person is putting into improving his/her favorite approach.

Well, I'm not in RT development anyway so that's not my case, but of course I'm could spend my time much better anyway ;)

Thanks again for your appreciated RT efforts Paul!

#73 lore raw
In reply to Ingo Molnar
Quoted 1 line · Ingo Molnar

Andrea, please just stop these scare tactics, it's getting boring. First

These are not scare tactics, infact if you use RTOS for something where a deadline has to be math guaranteed, and you certainly know that's not possible to guarantee a deadline with preempt-RT if the OS is under stress at the same time, and something goes wrong and I get a damage from it, be sure I'd consider you liable for recommending me to use solution that you know you can't demonstrate as correct, when fully reliable solutions were available and you knew it (for free as well as preempt-RT).

Quoted 3 lines

you came with incorrect statements about locks, then you came with the patent boogeyman, then you came with the driver bogosity, and now you come with "but ... [sputter] ... but ... where's the _guarantee_?".

I raised the "guarantee" argument in the second email I posted to this thread in explanataion of the statement "don't get the idea of using preempt-RT where hard-RT is required.". the staement "don't get the idea of using preempt-RT where hard-RT is required." comes from the very _first_ email I posted to this thread.

I don't see why you attempt to reverse the truth and you try to picture me as making up the next scare tactic of the day, when all posts are public and with timestamps and when I raised the "_guarantee_" at the very to of the thread, _before_ talking about _irq or local_irq_disable or the patent.

The reason I got into this thread is not some economic interest that some of the preempt-RT developers certainly has, nor any personal feeling about you. I've said it because I wouldn't like to get near a robot where the deadlines have to be met thanks to preempt-RT.

You've seen Bill answer when I've said I wouldn't even dream to use preempt-RT in the project that needs the 50usec latency right? That's the kind of thing that made me jump into this thread.

The rest of your email IMHO doesn't deserve comments except for this:

Quoted 3 lines

if you review the PREEMPT_RT technology (with a cool mind) i do think you'll eventually come to the conclusion that it is actually a pretty nifty idea and implementation, with some pretty good potential :-) It's

I've never stated otherwise, I've said _several_ times that in realtime apps like audio and multimedia in general, preempt-RT is certainly the best one can do if syscalls are involved anyways for a multitude of reasons. If you were a musician and you were asking me for a realtime solution I would sure point to preempt-RT, no argument about that. Even if local_irq_disable would still do an hard cli, that's still better than lowlatency alone, no? preempt can't get the irqs out of the way, preempt-RT can, and that will help audio if you've a usb modem like mine where the hardirq takes 8msec.

preempt-RT is a great feature at debugging as well, it is a clever design as well.

Quoted 3 lines

not trying to be the holy grail, it wont (nor does it want to) replace nanokernels, but it actually has some thinking behind it and is definitely useful to alot of people. And that is what matters. Your

This is the point I tried to make all the way long. Some people here clearly wants to use preempt-RT in all the domain of nanokernels and that's wrong IMHO (again see Bill's answer when I've said I wouldn't dream to use preempt-RT in the 50usec project on linuxdevices).

This is the source of all the flamewar as far as I can tell, and I'm glad that even despite your insults, at the very end you agree too.

#90 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Paul E. McKenney

f. Any code that manipulates hardware that can stall the
bus, delay interrupts, or otherwise interfere with forward progress. Note that it is also necessary to inspect user-level code that directly manipulates such hardware.

Quoted 6 lines
Quoted 1 line

I added point "f". Does that cover it?

Yes. F is really a bad problem if it really can cause DMA starvation on the memory bus on some arch as stated on this thread. Hopefully measurements are good enough to rule it out and there will be no corner cases triggering once in a while.

Agreed!

But who knows? Maybe we can convince some HW manufacturers to make their memory and I/O systems better. So it might also be good to have tests that forced the conditions as well as tests that avoid them. Though such tests do sound horribly hardware dependent...

Thanx, Paul

#32 lore raw
In reply to Andrea Arcangeli
Quoted 11 lines · Andrea Arcangeli
Quoted 2 lines · Paul E. McKenney

I just -know- I am going to regret stepping into the middle of this one...

Never mind, I keep regretting it myself. Perhaps it was better if I avoided posting here, and by now people would think preempt-RT was the holy grail of hard-RT and that would obsolete RTAI/rtlinux, when infact it's an inferior solution from from a reliability/guarantee standpoint and from a performance standpoint too (as some measurements showed already too) [and local_irq_disable would still be not emulated in software]

Well, I probably should have edited the "TO" and "CC" lines to make it clear that I was not singling you out. Please accept my apologies for making it appear that I was singling you out.

But at least we all know that both you and Bill care deeply about Linux! ;-)

Quoted 6 lines
Quoted 5 lines

All I have seen Ingo claim is that -some- hardware configurations running -some- specific workloads had excellent -measured- maximum scheduling latencies. I have not seen Ingo claim that he has mathematically proven that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let alone make any claim of what such a mathematically proven bound might be.

Quoted 8 lines

Indeed. As I stated in my first relevant post to this thread on "Tue, 31 May 2005 18:11:57 +0200" _before_ most of the other quotes that I received today, and before realizing the _irq part of the spin_lock_irq was already emulated in software and before realizing that the rtlinux patent doesn't forbid to emulate in software the _irq part of the spin_lock_irq because I'm not really good at reading all the patent claims but I focused only on the patent concept (and regardless there is apparently somebody who rightfully paid lawyers for research in that area):

CONFIG_PREEMPT_RT continues to be a bit of a learning experience for me as well. As to patent claims, don't get me started...

Quoted 22 lines

For the determinism, you could do what Ingo did so far, that is to "measure" but there's no way a "measurement" can provide an hard-RT guarantee. The "measure" way is great for the lowlatency patches, and to try to eliminate the bad-latencies paths, but it _can't_ guarantee a "worst-case-latency".

This is what I posted in my _second_ email to this thread. Of course nobody is quoting the above, they picked other later emails, but if you go back to the second email (and first relevant one) I posted here, you'll see this text at the end. Sorry for that mistake about the _irq part and the related patent claims that I'm not good at reading cause it's written by lawyers, but that mistake made no difference to the bottom line cause local_irq_disable was still disable the hard irqs without being emulated in software. I was reading local_irq_disable that wasn't touched (local_irq_disable is the _first_ thing I check in every RT patch on earth) and so I assumed no _irq disable was emulated. My bad but it didn't change the bottom line.

Now even local_irq_disable can be emulated in software (incidentally happened after my suggestions last week) but that still leaves the above issue in my second email unsolved.

Quoted 5 lines

It's not like I made up this "guarante" of worst case latency argument after realizing that _irq was emulated and after local_irq_disable has been emulated too, as somebody apparently seems to have described today by providing a list of quotes ignoring the above argument that was the core of my first relevant email to this thread.

Yep, I do remember several emails in the earlier thread that seemed to claim hard-realtime response. One might be able to create some sort of software tool that scanned the required code to produce such guarantee, but I have not heard of anyone actually doing this with any Linux-based realtime scheme.

Quoted 2 lines

If you check the timestamp, I mentioned the above core issue, way _before_ going into the _irq details.

This does indeed match my memory of the earlier thread.

Quoted 20 lines
Quoted 3 lines

Ingo -has- greatly reduced the amount of kernel code that must be inspected to guarantee realtime response, and this is a great accomplishment and a great contribution. I believe that his approach

Indeed.

Quoted 4 lines

is more than "good enough" for a great number of applications that are traditionally thought of as "hard realtime". But, unless I missed something, it is not mathematically proven, nor is it in any way "certified".

Yes, it could be that 99% of the app that traditionally claims to need hard-RT will be fully covered by preempt-RT (assuming the cpu can keep up with the slowdown), but when I talk about hard-RT I only think at those apps like the linuxdevices article that beats at 50usec. To me hard-RT is all about guaranteeing to provide a deadline, the rest I call it lowlatency patches. preempt-RT is obviously a very great improvement to the lowlatency soft-RT approach and I agree a lot of those apps thought to need hard-RT will be fine with it.

Quoted 3 lines

But one should evaluate if an hard-RT guarantee is truly needed or not before picking preempt-RT, the performance should be evaluated as well cause irq handling is much faster than a context switch.

Yep, much care is required when selecting a realtime OS approach, whether on Linux or any other OS. Seems that care in selecting hardware is required as well -- the PCI issue with X was a new one for me!

Quoted 1 line

The slowdown numbers posted so far are quite impressive as well.

Agreed!

Quoted 2 lines

Think that in some recent research I'm doing, I'm looking to make the semaphore a spinlock to boost performance, preempt-RT is the opposite ;)

;-)

Quoted 18 lines
Quoted 8 lines

Some of the approaches involving separate RTOSes -can- legitimately claim have mathematically proven worst-case scheduling latencies. For example, the dual-OS/dual-core approach can do so, particularly in the case where the "RTOS" is a tight loop written on bare metal in hand-coded assembly. One might consider this to be a trivial example, perhaps even a stupid example, but there are a lot of apps out there that use exactly this approach.

Quoted 9 lines

The migration-between-OSes approach, such as RTAI Fusion, might also be able to mathematically guarantee RTOS scheduling latencies. And, unlike the dual-OS/dual-core approaches, it does provide a single environment to applications. However, it still results in two separate OS instances to administer, and, as near as I can tell, the RTOS has to stick its hands so deeply into Linux's internals that it might as well be part of the Linux kernel. Unless I am missing something, any sort of bug in the Linux kernel has a reasonable chance of messing up the RTOS, since the RTOS must paw through Linux's tasks.

Quoted 2 lines

Yes. I'm a beliver in the dual-OS/dual-core model, as the only one who can reasonably simply guarantee a deadline.

If your realtime application needs every scrap of realtime response that the hardware can deliver, then I agree that the dual-OS/dual-core approach is very likely the best choice. But it depends on what OS services the realtime portion of the application needs. Beyond a certain point, it might be better to accept some probability of scheduling failure from Linux than to accept a perhaps-higher probability of bugs from a less-throroughly-tested RTOS. Or maybe not. To decide, it is necessary to look carefully at the details of the app, the RTOS, and Linux.

Quoted 6 lines

The single image RTOS has much larger pieces of complex code involved in the RT equation (in your list you didn't mention the mutex code itself), plus this codes like the scheduler changes all the time (it's not like the irq code that pratically never changes in stable cycles), and it slowsdown quite a bit as well at emulating infinite cpus and overscheduling at every contention.

Excellent point, am adding "mutex code" to the list. How does the following look?

e. Any code that holds a lock, mutex, semaphore, or other resource
that is needed by the code implementing your new feature, as well as the code that actually implements the lock, mutex, semaphore, or other resource.

Quoted 10 lines
Quoted 6 lines

Can we say that one of these approaches is definitely "good enough" for all reasonable Linux RT work, and that we should therefore stop working on the other approach?

I might well be missing something, but I don't believe so, at least not yet.

I agree with you. Perhaps I didn't make my point of view clear enough on the good things of preempt-RT. preempt-RT is a great feature for some apps.

Or I could easily have misread or misunderstood your emails.

Quoted 10 lines

I _only_ criticise people who wants to put preempt-RT on par with guaranteed hard-RT solutions like RTAI/rtlinux that can easily provide a guaranteed deadline, while preempt-RT can only proviede measurements backed by statistically signficance, which is quite far away from a guarantee. Plus it's much slower as well. But apparently some people here, wants you to believe that preempt-RT will be exactly guaranteed hard-RT as RTAI/rtlinux, and they can't handle it when I say this is not the case. This is the only thing I'm objecting as stated in my first relevant email of the thread partly quoted above.

Quoted 3 lines

I'm sure I've ever said that preempt-RT is useless and there's no application for it, infact I stated _multiple_ times there are cases like alsa where preempt-RT is the very best that one can do.

The big thing that I learned in reviewing the thread and in doing the writeup is that there are a lot of different things that a realtime app might want from a OS and from the hardware. It seems to me that we probably cannot do a simple linear ranking of the approaches -- some will be better at one aspect, others will be better at another.

Though I would love to be proven wrong! It is always nice from a user's point of view if you can just use the "best" one and spend more time worrying about getting your app right instead of having to spend a lot of time worrying about what OS approach to use! ;-)

Quoted 5 lines
Quoted 2 lines

In the meantime, the more time anyone spends sending flames on LKML, the less time that person is putting into improving his/her favorite approach.

Well, I'm not in RT development anyway so that's not my case, but of course I'm could spend my time much better anyway ;)

If you were -anywhere- at all near the worst example of misdirected effort, the pace of Linux development would be terrifyingly fast! ;-)

Quoted 1 line

Thanks again for your appreciated RT efforts Paul!

Glad you liked it, and, again, please accept my apologies for appearing to be singling you out.

Thanx, Paul

#33 lore raw
Quoted 5 lines · Paul E. McKenney

The big thing that I learned in reviewing the thread and in doing the writeup is that there are a lot of different things that a realtime app might want from a OS and from the hardware. It seems to me that we probably cannot do a simple linear ranking of the approaches -- some will be better at one aspect, others will be better at another.

During the earlier thread, I did suggest a third option which I think could help integrate both PREEMPT_RT and the interrupt pipeline in Adeos and RTAI-fusion _without_ impacting on the rest of the kernel code. However, I haven't received any feedback whatsoever in that regard. So here it is again. Given how much discussion space is taken on "concepts", it may be worth entertaining a third option.

Quoted 27 lines

<alternate proposal>
Much like there is nothing precluding PREEMPT_RT to co-exist with the nanokernel approach (on which RTAI is based), it could be suggested the adding of a linux/hard-rt directory containing the (re?)implementation of services/abstractions required for hard-rt applications. You still get a single tree, but there's then a clear separation at many levels, including maintainership. As such, much of what RTAI-fusion is currently doing could find itself in linux/hard-rt. For example, RTAI-fusion transparently provides a hard-rt deterministic nanosleep(). This and other such replacements for kernel/*.c would live in hard-rt/ with no disturbance to the rest of the tree. In the same way, include/linux could be a symbolic link to either include/linux-hrt or include/linux-srt, with headers in include/linux-hrt referring back to include/linux-srt where appropriate. Again, zero cost for mainstream maintainers. If the hard-rt stuff breaks, only the rt folks get the pain. Note that I'm not suggesting creating duplicates like this for all directories. In fact, most of what's in arch/* and drivers/* would remain unchanged, and where appropriate, hard-rt/* and include/linux-hrt should reuse as much of what already exists as possible.

Sure, the hard-rt part wouldn't have all the bells and whistles of the mainstream part, but that's what we're going to have anyway if PREEMPT_RT is included (as is clearly acknowledged elsewhere in this thread by those backing it), unless there's a general consensus amongst all subsystem maintainers that Linux should become QNX-like ... which, to the best of my reading of this thread, most are not interested in.

Quoted 5 lines

The above suggestion doesn't solve the two-app vs. one-app dilemma, but it takes away the "oh, horror, we need to maintain two separate kernel trees for our application development" from those against the nanokernel approach _without_ imposing additional burden on mainstream maintainers. </alternate proposal>

Let me clarify what I say above to make it clear that linux/hard-rt and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and rtai-fusion. The combo patch put together by Philippe (which includes both PREEMPT_RT and adeos) is already a good start in that direction. Like we said earlier, both methods are not mutually exclusive.

Note that my purpose by posting this proposal is to invite further discussion as to what is the best way to integrate any real-time approach to the existing kernel structure. What approach eventually makes it into this structure, whether it be PREEMPT_RT, Adeos, fusion, a combination of these, or something entirely different, is not something I attempt to resolve here. As it currently stands, for PREEMPT_RT at least, the intrusiveness of the patch onto the mainstream code is something that seems to make a lot of people uneasy.

Like I said earlier:

Quoted 3 lines

... so here goes, it's just an idea I'm throwing in the lion pit ... it clearly would require much more work and input ... so devour, tear, and crush at will ...

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#34 lore raw
Quoted 5 lines · Karim Yaghmour

Let me clarify what I say above to make it clear that linux/hard-rt and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and rtai-fusion. The combo patch put together by Philippe (which includes both PREEMPT_RT and adeos) is already a good start in that direction. Like we said earlier, both methods are not mutually exclusive.

FWIW I definitely agree that merging PREEMPT_RT design and nanokernel design in a single RT kernel makes lots of sense, and this confirms my original belief that preempt-RT and nanokernel are orthogonal and they don't cover the same domain of RT apps (or they'd overlap completely and it would be pointless to have them at the same time in the same kernel to use them at the same time).

#70 lore raw
In reply to Andrea Arcangeli
Quoted 3 lines · Andrea Arcangeli

I seriously hoped core things
like memory controller in the northbridge or embedded in the cpu would not risk to generate any starvation.

Nforce4 chipsets have some problems that look like DMA starvation too, the RME people claim to have traced it to the SATA controller.

http://www.rme-audio.de/english/techinfo/nforce4_tests.htm

Basically these all fall under common sense engineering, you don't use buggy/unproved hardware for an RT or any mission critical system.

Lee

#71 lore raw
Quoted 7 lines · Lee Revell

Nforce4 chipsets have some problems that look like DMA starvation too, the RME people claim to have traced it to the SATA controller.

http://www.rme-audio.de/english/techinfo/nforce4_tests.htm

Basically these all fall under common sense engineering, you don't use buggy/unproved hardware for an RT or any mission critical system.

How did folks work around that ? by using another SATA PCI-e card ?

bill

#69 lore raw
In reply to Paul E. McKenney
Quoted 3 lines · Paul E. McKenney

Well, I probably should have edited the "TO" and "CC" lines to make it clear that I was not singling you out. Please accept my apologies for making it appear that I was singling you out.

No need of apologies, your email was very appreciated. I greatly regret posting here only after reading the weird emails that some people from some other company posts. Having to pass emails through the politically-correct filter before sending them isn't very entertaining work.

Quoted 1 line

as well. As to patent claims, don't get me started...

Well, AFIK we're not supposed to be able to read them cause we're not lawyers, so I don't feel guilty in not understanding them completely, I try only to understand and remember the _concept_ of the patent, I don't even try to understand or remember all the legal details of the claims which seems completely unreadable for me even though I know it's the most important part of any patent.

Quoted 3 lines

sort of software tool that scanned the required code to produce such guarantee, but I have not heard of anyone actually doing this with any Linux-based realtime scheme.

With the nanokernel approach it's quite easy to provide a guarantee because there are no million of possible paths that the software can take. So demonstrating that with cold cache it takes less than 50usec to reach the RT userland regardless whatever path is taken is quite easy and feasible IMHO.

One still has the risk of the system crashing with memory corruption from a bad device driver, but the hardware can crash too. At some point one has to relay on stress testing, but starting from a weak design doesn't sound correct to me, especially if later something goes wrong in production (which from my point of view would make the designer liable if the design was flakey and the proper math wasn't done, it's like a bridge falling because proper math wasn't done, unless one tells the customer to use the stuff at his own risk, which doesn't sound a great form of "guarantee" in the first place ;).

With preempt-RT RTOS there are way too many possible paths to even attempt at providing a guarantee IMHO.

Quoted 1 line

is required as well -- the PCI issue with X was a new one for me!

I knew pci hardware can stall the bus, but I seriously hoped core things like memory controller in the northbridge or embedded in the cpu would not risk to generate any starvation. So far I thought by ruling out desktop graphics card or other fancy pci stuff that needs some extreme performance, one would have been fairly safe after some benchmarking. I hope on some embedded arch like arm people is careful to design northbridge to be RT compliant.

Quoted 8 lines

If your realtime application needs every scrap of realtime response that the hardware can deliver, then I agree that the dual-OS/dual-core approach is very likely the best choice. But it depends on what OS services the realtime portion of the application needs. Beyond a certain point, it might be better to accept some probability of scheduling failure from Linux than to accept a perhaps-higher probability of bugs from a less-throroughly-tested RTOS. Or maybe not. To decide, it is necessary to look carefully at the details of the app, the RTOS, and Linux.

Agreed.

I think using preempt-RT RTOS in problems where RTAI/rtlinux is equally simple would be a big waste and an unnecessary lower reliability. RTOS tries to keep the userland API as close as possible to the non-RT one, by increasing the kernel complexity with relative slowdown. So the whole system is certainly more complicated and slower even though the userland part alone may be a bit simpler.

Things becomes different if the RT app isn't fully in userland (like multimedia), once the RT app has to call deep into kernel code, then RTAI quickly becomes useless (ok it can run nanosleep with fusion fine, but I don't see how RTAI can run getdents or stuff like that which requires shared non-RT aware locks). Clearly it's possible to design a non-RT thread doing the non-RT stuff, and passing up the info to the RTAI userland RT thread, but RTOS is better at that, since the kernel locks will have RT awareness and because irqs will get out of the way.

But once somebody calls into the kernel for getdents, one can forget the word hard-RT, the mutex will get out of the way in the userland, but not in the kernel code, and kmalloc will swap.

Infact from my point of view preempt-RT and RTAI are fully orthogonal.

My view is still preempt-RT as a much much much improved soft-RT, and RTAI/rtlinux as the only hard-RT with guaranteed deadline.

If it was me having to call stuff like getdents in userland from a RT context, I could try using preempt-RT with a soft-RT thread passing up the getdents results to a RTAI hard-RT thread, mixing RTAI and preempt-RT in the same system.

As usual when I talk about hard-RT I think at the 1st kind of hard-RT where missing a deadline is unacceptable. As said before, the hard-RT type 2 is fine with current soft-RT linux too by just calling gettimeofday after the deadline was supposed to be met (or by an equivalent hack in the scheduler calling gettimeofday in kernel space and killing the task or sending a signal, whatever).

Quoted 7 lines

Excellent point, am adding "mutex code" to the list. How does the following look?

e. Any code that holds a lock, mutex, semaphore, or other resource
that is needed by the code implementing your new feature, as well as the code that actually implements the lock, mutex, semaphore, or other resource.

Looks fine thanks!

Quoted 2 lines

Glad you liked it, and, again, please accept my apologies for appearing to be singling you out.

Don't worry, I didn't feel singled out, and infact I greatly appreciated your contribution, thanks.

#35 lore raw
In reply to Karim Yaghmour
Quoted 8 lines · Karim Yaghmour
Quoted 5 lines · Paul E. McKenney

The big thing that I learned in reviewing the thread and in doing the writeup is that there are a lot of different things that a realtime app might want from a OS and from the hardware. It seems to me that we probably cannot do a simple linear ranking of the approaches -- some will be better at one aspect, others will be better at another.

Quoted 6 lines

During the earlier thread, I did suggest a third option which I think could help integrate both PREEMPT_RT and the interrupt pipeline in Adeos and RTAI-fusion _without_ impacting on the rest of the kernel code. However, I haven't received any feedback whatsoever in that regard. So here it is again. Given how much discussion space is taken on "concepts", it may be worth entertaining a third option.

Good point -- I did "fly over" this one on earlier reading, partly because I was somehow thinking that it was a dual-system-call-table approach so that realtime processes would see deterministic implementations of system calls (or get migrated/errored/whatever if there was no deterministic implementation for that particular system call), while non-realtime processes would see the standard stuff.

Quoted 27 lines
Quoted 27 lines

<alternate proposal>
Much like there is nothing precluding PREEMPT_RT to co-exist with the nanokernel approach (on which RTAI is based), it could be suggested the adding of a linux/hard-rt directory containing the (re?)implementation of services/abstractions required for hard-rt applications. You still get a single tree, but there's then a clear separation at many levels, including maintainership. As such, much of what RTAI-fusion is currently doing could find itself in linux/hard-rt. For example, RTAI-fusion transparently provides a hard-rt deterministic nanosleep(). This and other such replacements for kernel/*.c would live in hard-rt/ with no disturbance to the rest of the tree. In the same way, include/linux could be a symbolic link to either include/linux-hrt or include/linux-srt, with headers in include/linux-hrt referring back to include/linux-srt where appropriate. Again, zero cost for mainstream maintainers. If the hard-rt stuff breaks, only the rt folks get the pain. Note that I'm not suggesting creating duplicates like this for all directories. In fact, most of what's in arch/* and drivers/* would remain unchanged, and where appropriate, hard-rt/* and include/linux-hrt should reuse as much of what already exists as possible.

Sure, the hard-rt part wouldn't have all the bells and whistles of the mainstream part, but that's what we're going to have anyway if PREEMPT_RT is included (as is clearly acknowledged elsewhere in this thread by those backing it), unless there's a general consensus amongst all subsystem maintainers that Linux should become QNX-like ... which, to the best of my reading of this thread, most are not interested in.

Quoted 5 lines
Quoted 5 lines

The above suggestion doesn't solve the two-app vs. one-app dilemma, but it takes away the "oh, horror, we need to maintain two separate kernel trees for our application development" from those against the nanokernel approach _without_ imposing additional burden on mainstream maintainers. </alternate proposal>

This could potentially address the need for version-synchronization between RTAI-Fusion and the Linux kernel. Would you really want two separate builds, or is there some reasonable way of producing a single kernel binary that has both? And if there is some reasonable way of doing this, is it the right thing to do?

The single-binary approach could potentially reduce the dual-OS-administration load associated with RTAI-Fusion. However, handling all the interactions between the deterministic and non-deterministic system calls could get hairy. No big deal for scheduling primitives, but things could get interesting for I/O and networking protocols.

Quoted 5 lines

Let me clarify what I say above to make it clear that linux/hard-rt and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and rtai-fusion. The combo patch put together by Philippe (which includes both PREEMPT_RT and adeos) is already a good start in that direction. Like we said earlier, both methods are not mutually exclusive.

Agreed, most of the realtime approaches can be used together or combined.

Quoted 8 lines

Note that my purpose by posting this proposal is to invite further discussion as to what is the best way to integrate any real-time approach to the existing kernel structure. What approach eventually makes it into this structure, whether it be PREEMPT_RT, Adeos, fusion, a combination of these, or something entirely different, is not something I attempt to resolve here. As it currently stands, for PREEMPT_RT at least, the intrusiveness of the patch onto the mainstream code is something that seems to make a lot of people uneasy.

So, one can use the following types of combination:

o single source tree, multiple kernels (which is what I now
think that you are getting at above).

o straight merge, as between PREEMPT and PREEMPT_RT.

o single kernel, multiple syscall implementations for
some syscalls (deterministic vs. non-deterministic).

o side-by-side combination, as with dual-OS/dual-core and
pretty much any other approach.

Other combination approaches?

I have a paragraph in the updated version mentioning the possibility of combinations, and if some of these get "cooked" enough, will add more verbiage.

Quoted 4 lines

Like I said earlier:

Quoted 3 lines

... so here goes, it's just an idea I'm throwing in the lion pit ... it clearly would require much more work and input ... so devour, tear, and crush at will ...

Thanx, Paul

#36 lore raw
Quoted 5 lines · Paul E. McKenney

This could potentially address the need for version-synchronization between RTAI-Fusion and the Linux kernel. Would you really want two separate builds, or is there some reasonable way of producing a single kernel binary that has both? And if there is some reasonable way of doing this, is it the right thing to do?

No, single build is what I'm looking for. Nothing precludes the fusion parts from being built during the same kernel build ... as modules. If you don't load 'em, you don't need to worry about 'em.

Quoted 6 lines

The single-binary approach could potentially reduce the dual-OS-administration load associated with RTAI-Fusion. However, handling all the interactions between the deterministic and non-deterministic system calls could get hairy. No big deal for scheduling primitives, but things could get interesting for I/O and networking protocols.

Again, if you don't load 'em, you don't get 'em. If you use it and it's broken, then you're doing rt and you need to sync up with the maintainer. Nothing different here from the standard run of the mill "I'm using subsystem X and it doesn't work" posting to LKML.

Quoted 5 lines

So, one can use the following types of combination:

o single source tree, multiple kernels (which is what I now
think that you are getting at above).

Quoted 7 lines

o straight merge, as between PREEMPT and PREEMPT_RT.

o single kernel, multiple syscall implementations for
some syscalls (deterministic vs. non-deterministic).

o side-by-side combination, as with dual-OS/dual-core and
pretty much any other approach.

I'm not sure how you'd fit what I'm trying to suggest above, but let me rephrase it with the above in mind:

What I'm suggesting is that all rt components be included, but in separate directories within mainline. That may or may not mean additional schedulers/services. In the case where the new layout would include both PREEMPT_RT and fusion, what we'd get is that the user would have access to these configs:
- Plain Linux, no PREEMPT_RT, no ipipe, no fusion.
- Linux with PREEMPT_RT, no fusion: ints are threaded and locks are mutexes like now (however without the code intrusiveness given the use of separate directories.) May or may not include ipipe.
- Linux with fusion, no PREEMPT_RT: the fusion modules are built and installed with the rest of the modules. ipipe must be enabled.
- Linux with fusion and PREEMPT_RT: combination of the previous two.
- Linux with ipipe, no fusion or PREEMPT_RT: the soft-cli stuff is built into the kernel and loaded drivers can get deterministic response times, but there are no fancy rt services offered to anyone.

Practically, linux/hard-rt/ would contain both the code for PREEMPT_RT and the code for fusion. The actual layout in that directory would still need to be detailed, but the desired effect is that both PREEMPT_RT and fusion share as much code as possible.

Hope this clarifies what I'm suggesting a little bit more. Of course, all this would need to be rehashed a number of times, and most importantly, the PREEMPT_RT folks and the fusion effort would need to agree to join forces. From the fusion POV, it's clear that the door is open for collaboration. As proof, Philippe has been publishing combo patches with Adeos and PREEMPT_RT for some time. I can't speak for the PREEMPT_RT POV, though. I might be mistaken, but it seems that the feedback I've seen from some PREEMPT_RT backers does seem to indicate some openess. We'll see how things go.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#37 lore raw
Quoted 8 lines · Karim Yaghmour
Quoted 5 lines · Paul E. McKenney

This could potentially address the need for version-synchronization between RTAI-Fusion and the Linux kernel. Would you really want two separate builds, or is there some reasonable way of producing a single kernel binary that has both? And if there is some reasonable way of doing this, is it the right thing to do?

Quoted 4 lines

No, single build is what I'm looking for. Nothing precludes the fusion parts from being built during the same kernel build ... as modules. If you don't load 'em, you don't need to worry about 'em.

OK... Then the idea is to dynamically redirect the symbolic link to include/linux-srt or include/linux-srt that you mentioned in your previous email, or is the symlink serving some other purpose?

Quoted 7 lines
Quoted 6 lines

The single-binary approach could potentially reduce the dual-OS-administration load associated with RTAI-Fusion. However, handling all the interactions between the deterministic and non-deterministic system calls could get hairy. No big deal for scheduling primitives, but things could get interesting for I/O and networking protocols.

Quoted 5 lines

Again, if you don't load 'em, you don't get 'em. If you use it and it's broken, then you're doing rt and you need to sync up with the maintainer. Nothing different here from the standard run of the mill "I'm using subsystem X and it doesn't work" posting to LKML.

So your focus is on system calls that can have totally separate realtime and non-realtime implementations? Or am I missing some trick here?

Quoted 27 lines
Quoted 5 lines

So, one can use the following types of combination:

o single source tree, multiple kernels (which is what I now
think that you are getting at above).

Quoted 7 lines

o straight merge, as between PREEMPT and PREEMPT_RT.

o single kernel, multiple syscall implementations for
some syscalls (deterministic vs. non-deterministic).

o side-by-side combination, as with dual-OS/dual-core and
pretty much any other approach.

I'm not sure how you'd fit what I'm trying to suggest above, but let me rephrase it with the above in mind:

What I'm suggesting is that all rt components be included, but in separate directories within mainline. That may or may not mean additional schedulers/services. In the case where the new layout would include both PREEMPT_RT and fusion, what we'd get is that the user would have access to these configs:
- Plain Linux, no PREEMPT_RT, no ipipe, no fusion.
- Linux with PREEMPT_RT, no fusion: ints are threaded and locks are mutexes like now (however without the code intrusiveness given the use of separate directories.) May or may not include ipipe.
- Linux with fusion, no PREEMPT_RT: the fusion modules are built

Quoted 8 lines

and installed with the rest of the modules. ipipe must be enabled.
- Linux with fusion and PREEMPT_RT: combination of the previous two.
- Linux with ipipe, no fusion or PREEMPT_RT: the soft-cli stuff is built into the kernel and loaded drivers can get deterministic response times, but there are no fancy rt services offered to anyone.

Single kernel, multiple implementations for some syscalls, more or less, anyway.

Quoted 6 lines

Practically, linux/hard-rt/ would contain both the code for PREEMPT_RT and the code for fusion. The actual layout in that directory would still need to be detailed, but the desired effect is that both PREEMPT_RT and fusion share as much code as possible.

Quoted 10 lines

Hope this clarifies what I'm suggesting a little bit more. Of course, all this would need to be rehashed a number of times, and most importantly, the PREEMPT_RT folks and the fusion effort would need to agree to join forces. From the fusion POV, it's clear that the door is open for collaboration. As proof, Philippe has been publishing combo patches with Adeos and PREEMPT_RT for some time. I can't speak for the PREEMPT_RT POV, though. I might be mistaken, but it seems that the feedback I've seen from some PREEMPT_RT backers does seem to indicate some openess. We'll see how things go.

My guess is that there are enough people in the PREEMPT_RT camp that it might not make sense to ascribe a single point of view to them ;-)

How are you and Kristian looking to benchmark/compare the various combinations you call out above? Seems like one would have to look at more than straight scheduling/interrupt latency to make a reasonable comparison.

Thanx, Paul

#38 lore raw
Quoted 3 lines · Paul E. McKenney

OK... Then the idea is to dynamically redirect the symbolic link to include/linux-srt or include/linux-srt that you mentioned in your previous email, or is the symlink serving some other purpose?

What I'm suggesting is that rt patches shouldn't touch the existing codebase. Instead, functionality having to do with rt should be integrated in separate directories, and depending the way you configure the kernel, include/linux would point to either include/linux-srt or include/linux-hrt, much like include/asm points to one of inclux/asm-*.

Quoted 3 lines

So your focus is on system calls that can have totally separate realtime and non-realtime implementations? Or am I missing some trick here?

There's no trick. It's just a layout thing. Hope the above explains what I'm trying to say.

Quoted 4 lines

How are you and Kristian looking to benchmark/compare the various combinations you call out above? Seems like one would have to look at more than straight scheduling/interrupt latency to make a reasonable comparison.

I'm not sure that benchmarking would be relevant. This is just a integration/layout/configuration/build suggestion. I don't think that this organization will change anything to the benchmark results.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#39 lore raw
Quoted 11 lines · Karim Yaghmour
Quoted 3 lines · Paul E. McKenney

OK... Then the idea is to dynamically redirect the symbolic link to include/linux-srt or include/linux-srt that you mentioned in your previous email, or is the symlink serving some other purpose?

What I'm suggesting is that rt patches shouldn't touch the existing codebase. Instead, functionality having to do with rt should be integrated in separate directories, and depending the way you configure the kernel, include/linux would point to either include/linux-srt or include/linux-hrt, much like include/asm points to one of inclux/asm-*.

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

Daniel

#64 lore raw
In reply to Karim Yaghmour
Quoted 6 lines · Karim Yaghmour
Quoted 3 lines · Paul E. McKenney

OK... Then the idea is to dynamically redirect the symbolic link to include/linux-srt or include/linux-srt that you mentioned in your previous email, or is the symlink serving some other purpose?

Quoted 6 lines

What I'm suggesting is that rt patches shouldn't touch the existing codebase. Instead, functionality having to do with rt should be integrated in separate directories, and depending the way you configure the kernel, include/linux would point to either include/linux-srt or include/linux-hrt, much like include/asm points to one of inclux/asm-*.

I would guess that the end result would be a mixed strategy, where some things (e.g., the existing CONFIG_PREEMPT) are intermingled with the rest of the Linux code based, while other things (e.g., nanokernel implementations) are in separate directories. But this is quickly getting -way- outside of my area.

So I must leave this aspect of the discussion to others.

Quoted 6 lines
Quoted 3 lines

So your focus is on system calls that can have totally separate realtime and non-realtime implementations? Or am I missing some trick here?

There's no trick. It's just a layout thing. Hope the above explains what I'm trying to say.

OK. However, should the discussion get to the point where something like RTAI-Fusion has realtime versions of system calls that have globally-visible side-effects (such as I/O, networking, IPC, ...), then the issue of how to get the non-realtime and the realtime variants to play nicely with each other will arise.

It may well be that system calls containing such side-effects need to be Linux-only, or maybe someone will come up with the necessary tricks to make it all work nicely. Not particularly worried about it myself -- yet, anyway. There are plenty of things to worry about before we get to that point!

Quoted 9 lines
Quoted 4 lines

How are you and Kristian looking to benchmark/compare the various combinations you call out above? Seems like one would have to look at more than straight scheduling/interrupt latency to make a reasonable comparison.

I'm not sure that benchmarking would be relevant. This is just a integration/layout/configuration/build suggestion. I don't think that this organization will change anything to the benchmark results.

Sorry, side issue.

I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos, and Fusion, assuming (probably incorrectly) that you and Kristian were looking to compare all the possible combinations. If my assumption is incorrect, then my question was irrelevant, and I apologize for the noise.

Thanx, Paul

#40 lore raw
In reply to Daniel Walker
Quoted 4 lines · Daniel Walker

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

I understand what you're saying, but based on the feedback PREEMPT_RT has gotten up until now, and now outright suggestions that the debate is not even relevant to the LKML, I think that some people are trying to give those interested a hint: integration with mainline code is NOT on the agenda.

Some may want to continue trying to force-feed mainstream maintainers. I can't stop anyone from trying, that's for sure. However, I think what I'm suggesting is a reasonable compromise:
mainstream maintainers don't need to care about RT on a day-to- day basis and the RT folks get to be part of mainline.

That said, please make sure you've carefully read through what I suggest. I'm not saying that everything inside the kernel needs to be duplicated. I'm saying that what add-ons are should be separate. For example, like I was first saying, headers in include/linux-hrt would point back to include/linux-srt where appropriate.

And like I said earlier, this suggestion would need to be refined.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#41 lore raw
Quoted 11 lines · Karim Yaghmour
Quoted 4 lines · Daniel Walker

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

I understand what you're saying, but based on the feedback PREEMPT_RT has gotten up until now, and now outright suggestions that the debate is not even relevant to the LKML, I think that some people are trying to give those interested a hint: integration with mainline code is NOT on the agenda.

I think you're referring to one known flamer and hothead who most people on LKML don't seem to take seriously anymore. I suspect the silent majority does not mind as long as it's not intrusive on the current code base.

Lee

#62 lore raw
In reply to Daniel Walker
Quoted 13 lines · Daniel Walker
Quoted 11 lines · Karim Yaghmour
Quoted 3 lines · Paul E. McKenney

OK... Then the idea is to dynamically redirect the symbolic link to include/linux-srt or include/linux-srt that you mentioned in your previous email, or is the symlink serving some other purpose?

What I'm suggesting is that rt patches shouldn't touch the existing codebase. Instead, functionality having to do with rt should be integrated in separate directories, and depending the way you configure the kernel, include/linux would point to either include/linux-srt or include/linux-hrt, much like include/asm points to one of inclux/asm-*.

Quoted 4 lines

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

It depends on the details of the situation, right? For example, it would not have made sense to integrate RCU into spinlock.h, where most of the other Linux synchronization primitives live. On the other hand, spinlock variants, such as spin_lock_bh(), are integrated into the same spinlock.h file as the the base spin_lock() functions.

For example, CONFIG_PREEMPT_RT separates out some of its realtime function into separate files (e.g., kernel/rt.c and include/linux/rt_lock.h), while other parts of its realtime function are integrated into the rest of the kernel.

Thanx, Paul

#65 lore raw
In reply to Paul E. McKenney
Quoted 5 lines · Paul E. McKenney

OK. However, should the discussion get to the point where something like RTAI-Fusion has realtime versions of system calls that have globally-visible side-effects (such as I/O, networking, IPC, ...), then the issue of how to get the non-realtime and the realtime variants to play nicely with each other will arise.

Maybe so, but this will be a problem for the RT folks, not the mainstream folks, and that's why I believe this strategy is likely to be more feasible.

Quoted 4 lines

I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos, and Fusion, assuming (probably incorrectly) that you and Kristian were looking to compare all the possible combinations. If my assumption is incorrect, then my question was irrelevant, and I apologize for the noise.

Sorry, there's only so much we can do. Currently, we are redoing our earlier tests with what Ingo gave us.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#67 lore raw
In reply to Paul E. McKenney

On Mon, Jun 13, 2005 at 01:10:46PM -0700, Paul E. McKenney wrote:
...

Quoted 5 lines

It may well be that system calls containing such side-effects need to be Linux-only, or maybe someone will come up with the necessary tricks to make it all work nicely. Not particularly worried about it myself -- yet, anyway. There are plenty of things to worry about before we get to that point!

...

I suggest not speculating about the needs of various RT apps with respect to the kernel. (your speculation here will add more confusion to parts of an already confused kernel community) Vendors and other groups will layer them on as needed depending on specific usage. The futex/fusyn work is an exception with regard to syscalls, but don't speculate about projects like that if you haven't been carefully tracking their progress and their end goals.

That's my suggestion.

bill

#42 lore raw
In reply to Karim Yaghmour
Quoted 11 lines · Karim Yaghmour
Quoted 4 lines · Daniel Walker

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

I understand what you're saying, but based on the feedback PREEMPT_RT has gotten up until now, and now outright suggestions that the debate is not even relevant to the LKML, I think that some people are trying to give those interested a hint: integration with mainline code is NOT on the agenda.

I wouldn't work on RT if mainline integration wasn't on the agenda.

Quoted 5 lines

Some may want to continue trying to force-feed mainstream maintainers. I can't stop anyone from trying, that's for sure. However, I think what I'm suggesting is a reasonable compromise:
mainstream maintainers don't need to care about RT on a day-to- day basis and the RT folks get to be part of mainline.

There is going to be positive , and negative discussion on this. I think in the end the maintainers (Linus, and Andrew) don't want "people" to get a patch or modification from the outside. It's best if the community is not separated .. If you make a clean integration , and people want what you are doing, there is no reason for it to be rejected.

Daniel

#63 lore raw
In reply to Daniel Walker
Quoted 9 lines · Daniel Walker
Quoted 7 lines · Karim Yaghmour

...
What I'm suggesting is that rt patches shouldn't touch the existing codebase. Instead, functionality having to do with rt should be integrated in separate directories, and depending the way you configure the kernel, include/linux would point to either include/linux-srt or include/linux-hrt, much like include/asm points to one of inclux/asm-*.

Quoted 4 lines

I think this is mistake. Projects that create separation like this are begging for the community to reject them. I see this as a design for one, instead of design for many mistake. From what I've seen, a project would want to do as much clean integration as possible.

The problem with the patch at this time is that many of the headers, including the one you mention, need clean up as well as some reorganization. The problem with general sloppiness of the patch from rapid patch inclusion is different than the big event of integrating the patch into the mainline. Also, folks like me and you in our respective companies are much more aggressive about doing things with the patch which is the exception of Linux kernel users and not the norm. A little more testing wouldn't hurt as well, but I think the patch is certainly worth getting into an semi-experimental kernel.

IMO, the patch should go eventually, but after some significant clean up. There will be other areas in the kernel that will force some more proper abstraction and compartmentalization.

That's my view on the matter.

bill

#66 lore raw
In reply to Karim Yaghmour
Quoted 11 lines · Karim Yaghmour
Quoted 5 lines · Paul E. McKenney

OK. However, should the discussion get to the point where something like RTAI-Fusion has realtime versions of system calls that have globally-visible side-effects (such as I/O, networking, IPC, ...), then the issue of how to get the non-realtime and the realtime variants to play nicely with each other will arise.

Maybe so, but this will be a problem for the RT folks, not the mainstream folks, and that's why I believe this strategy is likely to be more feasible.

Unless correctly handling the side effects requires changes to the mainstream as well as to the RT implementations, as it might well for I/O, networking, and IPC. Again, I believe that it might be some time before we get to this point, so am not all that worried about it.

Quoted 7 lines
Quoted 4 lines

I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos, and Fusion, assuming (probably incorrectly) that you and Kristian were looking to compare all the possible combinations. If my assumption is incorrect, then my question was irrelevant, and I apologize for the noise.

Sorry, there's only so much we can do. Currently, we are redoing our earlier tests with what Ingo gave us.

Yours and Kristian's benchmarking contribution is quite timely and substantial, and I know that I am not the only one who very much appreciates it!

Thanx, Paul

#68 lore raw
In reply to Bill Huey (hui)
Quoted 7 lines · Bill Huey

On Mon, Jun 13, 2005 at 01:10:46PM -0700, Paul E. McKenney wrote:
...

Quoted 5 lines

It may well be that system calls containing such side-effects need to be Linux-only, or maybe someone will come up with the necessary tricks to make it all work nicely. Not particularly worried about it myself -- yet, anyway. There are plenty of things to worry about before we get to that point!

Quoted 6 lines

...

I suggest not speculating about the needs of various RT apps with respect to the kernel. (your speculation here will add more confusion to parts of an already confused kernel community) Vendors and other groups will layer them on as needed depending on specific usage.

Certainly good advice, but...

In this instance, I was speculating not about what various RT apps might need, but instead about what Karim was getting at. And I have seen cases where people needed realtime (as in deterministic) I/O, networking stacks, and the like, so that part is not speculation.

Whether or not deterministic implementations of I/O, networking stacks, and the like should ever be included in Linux certainly -is- speculation at this point, but, as I said in my earlier email, there are plenty of things to worry about before we get to that point.

Quoted 3 lines

The futex/fusyn work is an exception with regard to syscalls, but don't speculate about projects like that if you haven't been carefully tracking their progress and their end goals.

The futex/fusyn work is on my list, not for speculation, but rather for looking at what they have, what it does, how it fits in, and how it appears to be doing. My reason for looking into it is that I have also seen people saying that they need priority inheritance in user-level mutexes.

I would of course be quite interested in your thoughts on futex/fusyn!

Thanx, Paul

Quoted 5 lines

That's my suggestion.

bill

#43 lore raw
In reply to Daniel Walker
Quoted 1 line · Daniel Walker

I wouldn't work on RT if mainline integration wasn't on the agenda.

Mainline integration IS what I'm talking about. It's just not done the same way.

Quoted 5 lines

There is going to be positive , and negative discussion on this. I think in the end the maintainers (Linus, and Andrew) don't want "people" to get a patch or modification from the outside. It's best if the community is not separated .. If you make a clean integration , and people want what you are doing, there is no reason for it to be rejected.

I'm not suggesting the separation of the community, I'm suggesting a strategy of integration based on the fact that a large portion of kernel contributors don't necessarily care about RT, and most don't want to care about it in their day-to-day work (though I think most would care that Linux could have an additional spade down its sleeve, and would certainly try to help in as much they can from time to time.)

I'm not suggesting asking "people" to get patches from the outside. What I'm saying is that those developing mainstream code shouldn't need to worry about anything real-time, including modifications to locking primitives in headers (be they defined out or in).

In essence, what you ask can only hold if all kernel developers intend for Linux to become QNX. Clearly this isn't going to happen. Whatever changes are made to such core functionality as locking primitives and interrupt handling can hardly be "transparent" simply by wrapping #ifdef CONFIG_X around it in mainstream headers.

Quoted 1 line

From my point of view, determinism and best overall performance are

conflicting goals. Having separate derectories for something as fundamentally different from best overall performance as determinism is not too much to ask.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#44 lore raw
Quoted 1 line · Karim Yaghmour

I'm not suggesting the separation of the community, I'm suggesting

BTW, though I repeat that this wasn't what I was suggesting, it can be easily argued that community separation already exists. The fact of the matter is that few mainstream developers have been "asking" for RT. And at this point, some seem genuinely worried about its possible integration.

So one can keep ignoring the reality of the existing clash here, but it may just be that backing off a little and thinking about it in less absolute integration terms would help.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#45 lore raw
Quoted 5 lines · Karim Yaghmour

BTW, though I repeat that this wasn't what I was suggesting, it can be easily argued that community separation already exists. The fact of the matter is that few mainstream developers have been "asking" for RT. And at this point, some seem genuinely worried about its possible integration.

Ignore it. They are misinformed and ignorant about these issues.

Quoted 3 lines

So one can keep ignoring the reality of the existing clash here, but it may just be that backing off a little and thinking about it in less absolute integration terms would help.

Be a leader, not a follower. :)

You're never going to get anything cool done if you are "beat down" by easily offended folks that don't understand what you're doing.

bill

#46 lore raw
Quoted 1 line · Bill Huey (hui)

Ignore it. They are misinformed and ignorant about these issues.

Actually, what often happens on the LKML is that there a handfull who are ready to get flamed, and there's a pack in the back that would rather get some work done and have others take the heat for ideas they themselves share. So some may discount Andrea and Christoph, but I have a hunch there are others out there who think the same but just don't want to waste time feeding flames. ... after all, the patch isn't up for inclusion just yet ... :|

Quoted 1 line

Be a leader, not a follower. :)

Oh believe me, leading is not a problem here.

Quoted 2 lines

You're never going to get anything cool done if you are "beat down" by easily offended folks that don't understand what you're doing.

Been there done that. This is not about leadership, it's about equilibrium.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#47 lore raw
Quoted 2 lines · Karim Yaghmour

Been there done that. This is not about leadership, it's about equilibrium.

I'm tell'n you that there is no way that this patch is going to die. The commercial vendors and the general community is going use this once they figure out what it can do and there will be no stopping it. You can bet on some body extremity that this will happen. It's inevitable and ultimately the future of Linux whether they like it or not.

bill

#48 lore raw
Quoted 6 lines · Bill Huey (hui)

I'm tell'n you that there is no way that this patch is going to die. The commercial vendors and the general community is going use this once they figure out what it can do and there will be no stopping it. You can bet on some body extremity that this will happen. It's inevitable and ultimately the future of Linux whether they like it or not.

Please cool down. This is a rather extreme position.

The fact of the matter is that it is the community that is driving Linux, not the vendors. If the community decides something isn't going in, there isn't much any company can do about it, except maintain it outside of Linux for years on end, which economically speaking is just not feasible. That's why inclusion is so important.

Bare in mind that I wasn't even talking about whether or not PREEMPT_RT was making it in. I was only talking about the logistics of inclusion of any RT extension. Please people, get a grip, we're just trying to figure out the best way to do this stuff.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#49 lore raw
Quoted 4 lines · Karim Yaghmour

Bare in mind that I wasn't even talking about whether or not PREEMPT_RT was making it in. I was only talking about the logistics of inclusion of any RT extension. Please people, get a grip, we're just trying to figure out the best way to do this stuff.

The best way to do this stuff has already been outlined by Ingo himself, and as far as I know he's still the man leading the show. And no one objected to his strategy so far.

There is just no rush. It doesn't have to be integrated ASAP, and actually not before a while, and then certainly not in a disturbing way. Lots of things have to be cleaned up in the patch, and even in the current stock kernel for that matter, and Ingo even agreed to that.

So any discussion about merging of PREEMPT_RT into mainline is simply hand waving at this point. Let it evolve and resume this discussion when there is actually something being proposed for merging into mainline.

Nicolas

#50 lore raw
Quoted 2 lines · Nicolas Pitre

The best way to do this stuff has already been outlined by Ingo himself, and as far as I know he's still the man leading the show.

Touchy we are I see. Sorry to disappoint you, but I'm not interested in stealing anyone's show. I've got my own workload thank you very much.

Quoted 2 lines

And no one
objected to his strategy so far.

Objections have been voiced. Go back and read earlier postings.

Quoted 4 lines

There is just no rush. It doesn't have to be integrated ASAP, and actually not before a while, and then certainly not in a disturbing way. Lots of things have to be cleaned up in the patch, and even in the current stock kernel for that matter, and Ingo even agreed to that.

Sorry, but go back and check who's asking for ASAP. Then go back a

Quoted 3 lines · few e-mails and read back what I

So one can keep ignoring the reality of the existing clash here, but it may just be that backing off a little and thinking about it in less absolute integration terms would help.

Certainly you realize that "backing off" and "thinking" have little to do with any sort of "rush".

Quoted 4 lines

So any discussion about merging of PREEMPT_RT into mainline is simply hand waving at this point. Let it evolve and resume this discussion when there is actually something being proposed for merging into mainline.

So in your opinion, we should all put our brains on idle until someone posts a patch containing the words "please apply"?

Understand that I've got no inclination to going into a flamewar. I just made some rearrangement suggestion and all hell broke loose with people retreating to their defense positions reiterating POVs that had little to do with what I was suggesting.

You can disagree with what I'm saying, but please read the previous posts for context and don't put words in my mouth that I haven't said.

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#51 lore raw
Quoted 8 lines · Karim Yaghmour
Quoted 2 lines · Nicolas Pitre

The best way to do this stuff has already been outlined by Ingo himself, and as far as I know he's still the man leading the show.

Touchy we are I see. Sorry to disappoint you, but I'm not interested in stealing anyone's show. I've got my own workload thank you very much.

Wow... Did I really offend you so much?

I pretty know who you are, in case you forgot that we know each other, and your reading into my words that I somehow might have been suggesting that you would do that work yourself is rather amusing.

Quoted 4 lines
Quoted 2 lines

And no one
objected to his strategy so far.

Objections have been voiced. Go back and read earlier postings.

I read them. Admitedly some people are quite enthousiastic about PREEMPT_RT and maybe a little too anxious to see it appear into mainline. This is however not INgo's case AFAICS.

Quoted 10 lines
Quoted 4 lines

There is just no rush. It doesn't have to be integrated ASAP, and actually not before a while, and then certainly not in a disturbing way. Lots of things have to be cleaned up in the patch, and even in the current stock kernel for that matter, and Ingo even agreed to that.

Sorry, but go back and check who's asking for ASAP. Then go back a

Quoted 3 lines · few e-mails and read back what I

So one can keep ignoring the reality of the existing clash here, but it may just be that backing off a little and thinking about it in less absolute integration terms would help.

Thing is that there is just no existing clash. At least not yet. The whole thing is not ready for merging as Ingo said and therefore any discussion on merging issues are rather premature at this point.

Quoted 7 lines
Quoted 4 lines

So any discussion about merging of PREEMPT_RT into mainline is simply hand waving at this point. Let it evolve and resume this discussion when there is actually something being proposed for merging into mainline.

So in your opinion, we should all put our brains on idle until someone posts a patch containing the words "please apply"?

Exactly. How often has it been said on this very list and elsewhere:
"show me the patch"? In other words, there is no point discussing any merging issues when the project leader himself says it is not ready and has not posted any patch labeled as merge candidate. I somehow trust Ingo for posting patches when they are ready to be reviewed for integration into mainline +_if_ and _when_ it is time. Until then loads of changes may happen in the PREEMPT_RT code randering any present discussion on merging rather moot.

There is no gain into getting too excited about things. And this sentence is not directed at Karim in particular. I'm sure the concerned people will recognize themselves.

Nicolas

#52 lore raw
Quoted 6 lines · Nicolas Pitre
Quoted 2 lines · Karim Yaghmour

So in your opinion, we should all put our brains on idle until someone posts a patch containing the words "please apply"?

Exactly. [...]

... unless you wish to get involved in the development of that particular patch of course. ;-)

Nicolas

#53 lore raw
In reply to Nicolas Pitre
Quoted 1 line · Nicolas Pitre

Wow... Did I really offend you so much?

People have been edgy on these thread, so I read your response in light of that.

Quoted 3 lines

I pretty know who you are, in case you forgot that we know each other, and your reading into my words that I somehow might have been suggesting that you would do that work yourself is rather amusing.

Yes, I'd say we know each other quite well, hence my surprise at the reading I made of your tone. I'm glad that you've read this must amusement, we'll laugh it off next month at OLS :)

Quoted 3 lines

Thing is that there is just no existing clash. At least not yet. The whole thing is not ready for merging as Ingo said and therefore any discussion on merging issues are rather premature at this point.

I know Ingo's clearly stated that he isn't asking for it to be included at this point. This I don't deny. Ingo, however, is not alone in working on this, as you noted yourself ...

Quoted 8 lines

Exactly. How often has it been said on this very list and elsewhere:
"show me the patch"? In other words, there is no point discussing any merging issues when the project leader himself says it is not ready and has not posted any patch labeled as merge candidate. I somehow trust Ingo for posting patches when they are ready to be reviewed for integration into mainline +_if_ and _when_ it is time. Until then loads of changes may happen in the PREEMPT_RT code randering any present discussion on merging rather moot.

Ah, well then please read my proposals in light of the sub-topic that's been floating around in those threads regarding PREEMPT_RT and fusion being orthogonal and how they could be used together.

As for "show me the patch", posting patches to LKML is by definition asking for some measure of feedback. You're correct in stating that until a proposal is made for inclusion, things may change. Nothing precludes making suggestions though ...

In any case, I hope the above has clarified a few things. In the mean time, I'll get back to the load of unrelated stuff that I need to work on here.

Cheers,

Karim

Signature · 3 lines
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546
#59 lore raw
In reply to Karim Yaghmour
Quoted 16 lines · Karim Yaghmour
Quoted 1 line · Daniel Walker

I wouldn't work on RT if mainline integration wasn't on the agenda.

Mainline integration IS what I'm talking about. It's just not done the same way.

Quoted 5 lines

There is going to be positive , and negative discussion on this. I think in the end the maintainers (Linus, and Andrew) don't want "people" to get a patch or modification from the outside. It's best if the community is not separated .. If you make a clean integration , and people want what you are doing, there is no reason for it to be rejected.

Quoted 4 lines

I'm not suggesting the separation of the community, I'm suggesting a strategy of integration based on the fact that a large portion of kernel contributors don't necessarily care about RT, and most don't want to care about it in their day-to-day work

I would suggest that "don't want to care about it" already led and could lead to more bugs similar to the bugs discovered due to RT enchancements. But the bottom line here is that these bugs are almost always bugs in SMP kernel as well about which each kernel developer have to worry about.

Quoted 10 lines

(though I think most
would care that Linux could have an additional spade down its sleeve, and would certainly try to help in as much they can from time to time.)

I'm not suggesting asking "people" to get patches from the outside. What I'm saying is that those developing mainstream code shouldn't need to worry about anything real-time, including modifications to locking primitives in headers (be they defined out or in).

Quoted 5 lines

In essence, what you ask can only hold if all kernel developers intend for Linux to become QNX. Clearly this isn't going to happen. Whatever changes are made to such core functionality as locking primitives and interrupt handling can hardly be "transparent" simply by wrapping #ifdef CONFIG_X around it in mainstream headers.

Do you hardly working on the kernel patched with RT patch in other configurations than PREEMPT_RT? But I'm working and can;t report any outstanding issues/degradation. These changes _are_ "transperent" due to preemtp modes configurability. So what do you have behind "can hardly be "transparent" simply by wrapping #ifdef CONFIG_X around it"?

Quoted 5 lines
Quoted 2 lines

From my point of view, determinism and best overall performance are

conflicting goals. Having separate derectories for something as fundamentally different from best overall performance as determinism is not too much to ask.

configurability of the kernel gives you possibily to choose what you want in your custom case. Source code organization is a matter of whatever else (readability, complexity, separation of completely different functionality (but this is not this case), desire for community acceptance, etc) but not that determinism and best overall performance are conflicting goals. Why don;t you object against SMP though it definitly could conflict with goals of desktop kernel configurations?

Eugeny

Quoted 1 line

Karim

#60 lore raw
In reply to Karim Yaghmour
Quoted 13 lines · Karim Yaghmour
Quoted 1 line · Daniel Walker

I wouldn't work on RT if mainline integration wasn't on the agenda.

Mainline integration IS what I'm talking about. It's just not done the same way.

Quoted 5 lines

There is going to be positive , and negative discussion on this. I think in the end the maintainers (Linus, and Andrew) don't want "people" to get a patch or modification from the outside. It's best if the community is not separated .. If you make a clean integration , and people want what you are doing, there is no reason for it to be rejected.

Quoted 7 lines

I'm not suggesting the separation of the community, I'm suggesting a strategy of integration based on the fact that a large portion of kernel contributors don't necessarily care about RT, and most don't want to care about it in their day-to-day work (though I think most would care that Linux could have an additional spade down its sleeve, and would certainly try to help in as much they can from time to time.)

It is probably worth pointing out that when companies like IBM and SGI (and many others that I don't want to slight) committed to making Linux scalable, they faced very similar issues to what you are facing now. Specifically, the community at that time (and even in many cases now) were not concerned about scalability _at all_.

Hence, the burdn was on those advocating scalability to find ways that minimized the code impact, that minimized the need for distinct CONFIG options, that allowed distros to build "one version" which worked everywhere, and at the same time simplified, or, at least, did not add substantial complexity.

I believe that any effort towards mainline support of RT has to follow a similar set of guidelines. And, I believe strongly that *most* of the RT code should be crafted so that every single laptop user is running most of the code *and* benefiting from it. If most of the RT code goes unused by most of the population, and the only way to get an RT kernel of any reasonable level is to ask the distros to build yet another configuration, RT will always be a poor, undertested, underutilized ugly stepchild of Linux.

And, in fact, RT has some goals which are very much in common with many Linux users. Simply the focus on audio and video playback and record capabilities and the realtime needs for those common applications should provide the basis for commonality in approach and implementation of RT code in the current mainline.

Of course, this leaves a not inconsiderable burden on the RT community to bridge their knowledge with the overall community goals of simplicity, code cleanliness, comprehensibility, maintainability, etc. But such an approach has a greater chance of success than replicating a "kernel in a kernel" approach where a CONFIG option enables an entirely different set of RT based characteristics from the mainline kernel that is most widely used.

Quoted 4 lines

I'm not suggesting asking "people" to get patches from the outside. What I'm saying is that those developing mainstream code shouldn't need to worry about anything real-time, including modifications to locking primitives in headers (be they defined out or in).

I would disagree a bit. Making it *simple* to give the kernel RT behavior in more and more subsystems will over time work much like scalability does today. If patches are agressively vetted, reviewed, tested, and commented on as they come in, e.g. via the -mm tree, it is possible to drive common development towards better and better RT
behavior. Of course, this requires simple APIs, lots of education,
occasional tools like sparse, or simple patches to measure latency as has been done many times in the past.

Quoted 5 lines

In essence, what you ask can only hold if all kernel developers intend for Linux to become QNX. Clearly this isn't going to happen. Whatever changes are made to such core functionality as locking primitives and interrupt handling can hardly be "transparent" simply by wrapping #ifdef CONFIG_X around it in mainstream headers.

Not all kernel developers need to subscribe to any philosophy, ever. However, by making it simple to validate patches for the capabilities you wish to advocate, it *is* possible to reject patches which detract from your capabilities or impede your ability to deploy those capabilities. And, when rejecting such patches, the burden will fall to the RT community to show *why* the patch won't work, as well as help identify better means for accomplishing the original patches intent, while also enabling better RT support.

No, this is not an easy path. But I believe it is a path towards the ultimate success of Linux being able to provide increasingly better RT support over time.

Quoted 4 lines

From my point of view, determinism and best overall performance are conflicting goals. Having separate derectories for something as fundamentally different from best overall performance as determinism is not too much to ask.

I remember a day when many claimed that desktop responsiveness and scalability were conflicting goals. I don't see those people as active any more and further, I note that over time, scalability improvements have in many cases furthered desktop responsiveness as well. You've taken the first step which is to identify the apparent conflict of goals. The next step is to find solutions which actually can address both of those goals, and do it with simple, clean, easy to maintain code. Not easy, but I know there are a lot of smart people out here who can rise to that particular challenge. ;)

gerrit

#54 lore raw
In reply to Karim Yaghmour
Quoted 9 lines · Karim Yaghmour
Quoted 6 lines · Bill Huey (hui)

I'm tell'n you that there is no way that this patch is going to die. The commercial vendors and the general community is going use this once they figure out what it can do and there will be no stopping it. You can bet on some body extremity that this will happen. It's inevitable and ultimately the future of Linux whether they like it or not.

Quoted 7 lines

Please cool down. This is a rather extreme position.

The fact of the matter is that it is the community that is driving Linux, not the vendors. If the community decides something isn't going in, there isn't much any company can do about it, except maintain it outside of Linux for years on end, which economically speaking is just not feasible. That's why inclusion is so important.

Everyone scratches their own itches - and if a vendor doesn't *become* a member of the community, including understanding the ground rules, the motivators and de-motivators, and does not become an advocate for those community goals, they will always be on the outside looking in. Being a member of the community can cost you your vendor soul. I really like the phrasing that Andrew used in his keynote at OLS last year. Reread this and think hard about it:

http://www.zip.com.au/~akpm/linux/patches/stuff/ols-2004.txt

In particular, I think this portion is key:

Well, I think a similar process is coming into play as corporate programmers are assigned to sit in their cubicles and work on
Linux. They come in as good little corporate people but some
of them are subverted. We end up taking over their brains
and owning them. They become members "of the community". They become Free software developers.

While this is a statement of what is happening, I think it is also an important statement about what needs to happen for the RT community to become part of the mainline development community. Until you have been subverted, you are going to keep talking about how to advocate *your* agenda. You need to advocate the *communities'* agenda, as a member of the subverted community who holds onto *all* of the major driving forces and goals for Linux.

Quoted 4 lines

Bare in mind that I wasn't even talking about whether or not PREEMPT_RT was making it in. I was only talking about the logistics of inclusion of any RT extension. Please people, get a grip, we're just trying to figure out the best way to do this stuff.

It's easy:

1) Change your allegiance to Linux first
2) Do the right thing.

And, implicit in that is that I believe that RT for the community will in the long term be The Right Thing.

gerrit

#61 lore raw
In reply to Gerrit Huizenga
Quoted 2 lines · Gerrit Huizenga

RT code should be crafted so that every single laptop user is running most of the code *and* benefiting from it. If most of the RT code goes

The problem is that it's not feasible to enable it by default and pretend not to hurt the larger part of the userbase.

Only the musicians and the videogame players may get a benefit from preempt-RT (probably the videogames players may not get a noticeable benefit either), everyone else would be hurt somehow.

I don't think it's correct to compare this to the smp scalability effort, where even a small 2-way was getting benefit from every SMP optimization done for a 16-way.

Also don't forget today UP systems still run with CONFIG_SMP disable to keep performance of those systems higher (that may change once UP systems becomes obsolete, but not because CONFIG_SMP is useless per se ;).

The RT design emulates infinite cpus, with the associated locking contention that infinite cpus would generate (actually the number is bound at some point, it's not really infinite but it depends on the hardware/software combination). To make things even worse every contention (even if short) generates overscheduling. So the performance impact of some workload is probably going to be very measurable for a small system that has no need of dealing with the infinite-cpus locking. Some slowdown number have been posted already, but probably debug options were enabled and there's room for further optimization, they seemed excessive bad numbers so far which can be improved, but no idea how much.

The mouse going faster kind of thing must be always verified with a placebo to give it any statistical value (if it cannot be measured), otherwise by applying the new feature it'll always go faster with the new feature applied, it happened to me as well sometime, humans aren't very reliable since our (or at least my) perception of time vary depending on the feelings.

To make an example I'm 100% sure that I would only get downsides by running preempt-RT in all my production systems (including laptops, desktop and servers), so while I may run it on my test systems to help testing, I would never consider to enable it on any of my production systems where I try to keep an optimal setup. This has never been the case with the smp scalability effort, I've always got benefits in all my smp systems, and I had CONFIG_SMP=n in the UP systems.

This is why there's a config option for preempt-rt I think, which makes perfect sense to me and I doubt it'll be enabled by default, at least unless they find a way to reduce the slowdown to not-measurable levels, which I strongly doubt, especially due the irq scheduling thingy.

Perhaps the only thing that I could benefit from is such horrible 8msec latency of the usb irq in my firewall, but that's a bug that needs fixing by offloading it to softirq or in some other way, preempt-RT would only hide the real problem at the expense of overscheduling.

non-RT irqs are always supposed to be as quick as possible and to do the processing in normal kernel context after a wakeup (or softirq context after posting a tasklet or raising a softirq, or even a kevent).

Overall I doubt preempt-RT config option objective was to be enabled by default in the kernel that gets installed on laptops, but I could be wrong.

#55 lore raw
In reply to Gerrit Huizenga
Quoted 4 lines · Gerrit Huizenga

Everyone scratches their own itches - and if a vendor doesn't *become* a member of the community, including understanding the ground rules, the motivators and de-motivators, and does not become an advocate for those community goals, they will always be on the outside looking in.

...

I agree. You're one of the few folks from IBM and other large companies to understand this. There's a real lack of technical leadership in this community and it is a fear of change in general as well as to this patch because of what seem to be an inability to think through various problems like this and be comfortable with it. That's why I went to FreeBSD for a while since the Linux VM system back in the day was doing nothing be thrashing. Even in 2005 it's still got problems relative to the BSDs for load handling it seems and this why I still track projects like DragonFly BSD since they are much more friendly to heavy design changes unlike Linux.

Quoted 7 lines

While this is a statement of what is happening, I think it is also an important statement about what needs to happen for the RT community to become part of the mainline development community. Until you have been subverted, you are going to keep talking about how to advocate *your* agenda. You need to advocate the *communities'* agenda, as a member of the subverted community who holds onto *all* of the major driving forces and goals for Linux.

In my case, I was already subverted by Linux and other open source projects. I just happen to get hired by an RTOS vendor to continue this work in some way.

Quoted 7 lines

It's easy:

1) Change your allegiance to Linux first
2) Do the right thing.

And, implicit in that is that I believe that RT for the community will in the long term be The Right Thing.

This implications of this patch are huge. It's transparent and will ultimately change the userspace land scape for just about everything despite the paranoia and disinformation spread by various arch-conservatives in this community.

bill

#56 lore raw
Quoted 3 lines · Bill Huey

This implications of this patch are huge. It's transparent and will ultimately change the userspace land scape for just about everything despite the paranoia and disinformation spread by various arch-conservatives in this community.

Is this patch running for political office? :)

#58 lore raw
In reply to Bill Huey (hui)
Quoted 6 lines · Bill Huey
Quoted 5 lines · Gerrit Huizenga

It's easy:

1) Change your allegiance to Linux first
2) Do the right thing.

Quoted 6 lines
Quoted 2 lines

And, implicit in that is that I believe that RT for the community will in the long term be The Right Thing.

This implications of this patch are huge. It's transparent and will ultimately change the userspace land scape for just about everything despite the paranoia and disinformation spread by various arch-conservatives in this community.

Hmm. I think a few people have misunderstood one aspect of my position here which bears clarification:

I'm making __NO__ statement on the validity of the specific RT patch under circulation. Well, no statement on whether it should go into mainline as it is. I think it is a cool patch, from what I've seen and read, it is way better than most things that have been kicked around.

However, what I __AM__ saying is: Linux over time can tolerate changes which will improve its ability to be used in various levels of Real Time situations. Those improvements may start out small, such as analyses of latency problems, or locking improvements and such. However, the thing about understanding the community is to figure out *which* solutions are the right solutions that move things closer to real time. If this patch fits all of the community goals and guidelines for simplicity, easy to maintain, clear, and evolves in the right general direction, then, yes, it is probably the right patch. But even if it isn't, that doesn't mean the end of RT support for Linux - it just means that the community needs to continue to collaborate in the right direction and this particular patch can be used as a yardstick for progress in some cases.

This isn't about defending a particular patch to the death (even if it is the best one we've seen so far) but about evolving solutions to resolve the core problem while not breaking anything else. This patch *might* be the next step in that evolution. It *looks* like it could be a good step in the right direction and useful to test in -mm or the like for a while. But odds are it will also introduce some problems and need some continued evolution as well.

gerrit

#57 lore raw
In reply to [email protected]
Quoted 7 lines · On Tuesday 14 June 2005 15:35, [email protected] wrote:
Quoted 4 lines · Bill Huey

This implications of this patch are huge. It's transparent and will ultimately change the userspace land scape for just about everything despite the paranoia and disinformation spread by various arch-conservatives in this community.

Is this patch running for political office? :)

That's entirely possible Valdis. This patch has intrigued me for almost all the time that Ingo has been working on it, and I'm currently running 48-19. Now he has converted a few others to both help and guide. This cannot be anything but a plus for linux in the long view. Unforch, I see, call it from the vantage point of an elder geek or whatever viewpoint, the digging in of heels and the speechifying against the changes proposed by those that in some sense consider themselves the 'guardian of the flame' or something.

I am somewhat reminded of Peter the Great, whose statement at the time was that he was going to bring Russia into the 20th century, kicking
and screaming if neccessary. I guess this is the kicking and
screaming phase?

But, I also see this as a sign of health because I've read some pretty decent and very educational arguments here (I even half understood some of them!), with only minimal name-calling (I guess you all reserve that for JS maybe?) and have seen a few change their mind to become 'supporters of the cause'.

So lets all just keep on keeping on. This WILL sort itself given enough time to get the application programmers onboard. It is they who will find that they are the ones who need this, but they will also need to learn how to code to take advantage of it to its fullest. For that, it needs a stable interface. But its only going to become 'stable' by going mainline so it gets enough exposure and everything gets sorted eventually.

A sample app that needs help badly is kmail, any background processes involved with incoming mail sorting and filtering result in a frozen user interface for the duration of the background process. So any kde people copying the mail here, sic 'em! My guess is that this is exactly what you need to remove that particular user headache/itch. That's one of my personal itches...

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.