PREEMPT_RT vs I-PIPE: the numbers, part 2
We've finally been able to complete a second round of our tests. Unfortunately, this has taken much longer than we ever anticipated. Without going into too much detail, suffice it to say that we ran into problems with each of our intended test configurations. Fortunately, the results are that much more interesting and, we hope, more accurately illustrate the performance of the approaches being compared.
The setup is exactly the same as the one previously described. So if you've missed our earlier posting, it'll probably be easier to go read that one first for background info:
http://marc.theaimsgroup.com/?l=linux-kernel&m=111846495403131&w=2
Of course we would have liked to post the complete packages and configs that were used instead. Given the time spent on getting the tests to work again with the new configs, we were unable to put together a decent package though. Without further promising any dates, we will make those available shortly.
Also, note that the versions of the approaches we were testing went through significant changes between those versions we last used and the versions published in the current testset.
For PREEMPT_RT, word from Ingo is that a lot of things were fixed since the version we were using in our last tests. Hence, we used a more recent version and relied on the .config provided to us by Ingo. And indeed the numbers seem to confirm Ingo's analysis. As we will see below, PREEMPT_RT behaves quite well.
For Adeos ... hmm Adeos? what's that ... I mean I-pipe ... well that's the news about it isn't it. While we were working on benchmarking his stuff, Philippe refactored the core component of the Adeos patch in order to isolate the interrupt pipeline therein implemented. Philippe's recent posting of these patches should provide a better explanation than we can put here. But basically, what we used and measured in Adeos is very much the same thing we are doing with the I-pipe here. Though the version posted Friday by Philippe to the LKML is more recent than the one we used for our tests, it should be good enough for comparison.
Another change since the last test run is that in this case, both setups are compared to the same kernel, 2.6.12-rc6. Needless to say that we are much happier with comparing two approaches to the same exact kernel version instead of trying to correlate results from two different kernels.
In terms of changes to the tests, a few minor things have changed. Most importantly, the hd test now correctly contains a "bs=1m", and hence results in the creation of a 512MB file on disk. Unlike the wishes we expressed earlier, we haven't integrated additional tests to the ones we had already carried out. It was enough work already to get a repeat of the same tests that followed the recommendations of the proponents of each method. Those tests that were mentioned remain relevant nonetheless, especially hackbench and dbench.
So here are the results, and an attempted analysis. As was said earlier, we don't believe any single test run will ever definitively rule in favor or against either approach. Only continued benchmarking will help steer discussions we hope in the right direction.
Total system load:
------------------
Like last time, total system load is measured by LMbench's total execution time under various system loads. Again, these are on an average of 3 runs each, so the previous caveats still apply:
on such a few runs, the numbers should be read as a general tendency with more definitive numbers requiring many more repeats.
LMbench running times:
+--------------------+-------+-------+-------+-------+-------+ | Kernel | plain | IRQ | ping | IRQ & | IRQ & | | | | test | flood | ping | hd | +====================+=======+=======+=======+=======+=======+ | Vanilla-2.6.12-rc6 | 175 s | 176 s | 185 s | 187 s | 246 s | +====================+=======+=======+=======+=======+=======+ | with RT-V0.7.48-25 | 182 s | 184 s | 201 s | 202 s | 278 s | +--------------------+-------+-------+-------+-------+-------+ | % | 4.0 | 4.5 | 8.6 | 8.0 | 13.0 | +====================+=======+=======+=======+=======+=======+ | with Ipipe-0.4 | 176 s | 179 s | 189 s | 190 s | 260 s | +--------------------+-------+-------+-------+-------+-------+ | % | 0.5 | 1.7 | 2.2 | 1.6 | 5.7 | +--------------------+-------+-------+-------+-------+-------+
Legend:
plain = Nothing special IRQ test = on logger: triggering target every 1ms ping flood = on host: "sudo ping -f $TARGET_IP_ADDR" IRQ & ping = combination of the previous two IRQ & hd = IRQ test with the following being done on the target:
"while [ true ]
do dd if=/dev/zero of=/tmp/dummy count=512 bs=1m
done"
In general, it seems that rc6 provides similar results as rc2 and rc4 did in our earlier tests, so that's a good indication of the validity of those numbers. The only number that is significantly different is the IRQ & hd test, but that is easily explained by the fact that in our earlier test we did not have the "bs=1m" in the "dd" command. Hence, only a 512 bytes file was generated instead of the intended 512MB.
For PREEMPT_RT the numbers are also similar to those we found last time, which again helps confirm the tendency. However, as expected, the HD test is significantly different, for the same reasons as above (namely "bs=1m"). Without entering the world of speculation too much, it should be fair to say that the overhead generally observed in the various PREEMPT_RT runs in comparison to the vanilla runs is likely due to the additions introduced by the patch to some of the kernel's critical mechanisms.
For the Ipipe, the numbers are better than those observed for Adeos in the last run for the light loads, but not as good for the heavy loads. While the I-pipe's impact remains much lower than PREEMPT_RT, as was the case for Adeos in the last test run, it is rather difficult to compare both test results as the I-pipe is not exactly Adeos. One important modification is that Adeos domains each ran using a separate stack and Adeos switched to the appropriate stack when switching domains. The current I-pipe has had this functionality stripped from it and, for what we make of it, the interrupts are handled on whatever stack exists at that time. It appears that this had the advantage of reducing the size of the patch, but if the I-pipe can be compared to Adeos, then it appears that removing this functionality has reduced performance. It is also possible that the slowdown may be due to subtle bugs introduced by the refactoring. More testing would need to be carried out to better determine the cause.
In general, it appears that I-pipe's impact on general system performance is lower than PREEMPT_RT's.
Interrupt response time:
------------------------
Like last time, interrupt response time is measured by the delay it takes for the target to respond to interrupts from the logger. Given that somewhere between 500,000 and 650,000 interrupts are generated by the logger for each test run, we believe that these results illustrate fairly well the behavior of the measured approaches.
Time in micro-seconds:
+--------------------+------------+------+-------+------+--------+ | Kernel | sys load | Aver | Max | Min | StdDev | +====================+============+======+=======+======+========+ | | None | 13.9 | 55.5 | 13.4 | 0.4 | | | Ping | 14.0 | 57.9 | 13.3 | 0.4 | | Vanilla-2.6.12-rc6 | lm. + ping | 14.3 | 171.6 | 13.4 | 1.0 | | | lmbench | 14.2 | 150.2 | 13.4 | 1.0 | | | lm. + hd | 14.7 | 191.7 | 13.3 | 4.0 | +--------------------+------------+------+-------+------+--------+ | | None | 13.9 | 53.1 | 13.4 | 0.4 | | | Ping | 14.4 | 56.2 | 13.4 | 0.9 | | with RT-V0.7.48-25 | lm. + ping | 14.7 | 56.9 | 13.4 | 1.1 | | | lmbench | 14.3 | 57.0 | 13.4 | 0.7 | | | lm. + hd | 14.3 | 58.9 | 13.4 | 0.8 | +--------------------+------------+------+-------+------+--------+ | | None | 13.9 | 53.3 | 13.5 | 0.8 | | | Ping | 14.2 | 57.2 | 13.6 | 0.9 | | with Ipipe-0.4 | lm.+ ping | 14.5 | 56.5 | 13.5 | 0.9 | | | lmbench | 14.3 | 55.6 | 13.4 | 0.9 | | | lm. + hd | 14.4 | 55.5 | 13.4 | 0.9 | +--------------------+------------+------+-------+------+--------+
Legend:
None = nothing special ping = on host: "sudo ping -f $TARGET_IP_ADDR" lm. + ping = previous test and "make rerun" in lmbench-2.0.4/src/ on
target
lmbench = "make rerun" in lmbench-2.0.4/src/ on target
lm. + hd = previous test with the following being done on the
target:
"while [ true ]
do dd if=/dev/zero of=/tmp/dummy count=512 bs=1m
done"
In last week's test run it may have seemed that there was a distinct pattern for each setup and that average response times were slightly different (give or take a microsecond here or there.) In the current test run it appears that the average response time in all configurations is identical, that the same can be said about the minimum response times, and that vanilla clearly has much larger maximum response times when compared to either real-time extension.
For PREEMPT_RT clearly the results are much better than last time. Indeed it appears that, as Ingo predicted, a combination of the proper configuration options and most recent additions gives PREEMPT_RT important gains. In comparison to last week's results all measures are lower: average response time, maximum response time, minimum response time, and standard deviation. This is very good. But that's not all. PREEMPT_RT also comes down virtually neck-to- neck with the I-pipe (and the previous numbers from Adeos) in terms of maximum interrupt response time. Certainly those backing PREEMPT_RT, and others we hope, will find this quite positive.
The I-pipe, for its part, has yielded overall identical results to Adeos' results from last week. In doing so, it confirms its claims of inheriting Adeos' most important feature: the ability to obtain deterministic interrupt response times.
Overall analysis:
-----------------
The guiding principal in carrying out those tests has been to help us, and we hope others, understand the impact the proposed real-time additions have on Linux. As such, these numbers are far from being the entire story. They only provide additional hints to those studying Linux's progression towards real-time responsiveness.
Clearly the approaches analyzed here take a different path to their enabling of real-time in Linux. And while both are being submitted by separate groups to the general Linux community, it is important, as was made clear in earlier threads, to highlight that these approaches can and have already been used together -- some have outright labeled them orthogonal. It follows that a comparison between these approaches should not necessarily be used for trying to determine a "winner".
Instead, we suggest that the above tests will likely be used to better help those needing real-time responsiveness to decide which approach is best for them. As can be expected, such choices are often guided by compromise. We would have liked to offer some general guidelines, but after thinking long and hard, we thought it'd be best if those were laid out through the _constructive_ criticism of the community. Nonetheless, here are some general _comments_ to start things off.
On the face of it, one would be tempted to conclude that if you are looking for a lightweight mechanism for obtaining deterministic interrupt response times, then the I-pipe seems to be a pretty safe bet. After all, its impact on general system performance seems less than that of PREEMPT_RT.
However, the I-pipe alone doesn't replace the functionality offered by PREEMPT_RT, most importantly in regards to the rt scheduling of user-space processes.
So at this point one would be tempted to conclude that if you're looking for a fully integrated rt solution for Linux, PREEMPT_RT would be the best candidate. After all it seems to provide as good interrupt response times as I-pipe while providing rt scheduling of user-space processes, albeit with a price on general system performance.
But as was stated earlier in these rt threads -- then the argument was made for Adeos, but it still applies for the I-pipe --, the I-pipe is also an enabler for running an rt executive side-by-side with Linux. With Fusion, for example, user-space processes can be provided with rt scheduling and other services using the transparent migration of service requests between Linux and the RT-executive.
Again, as we said earlier, we do not believe that the approaches are mutually exclusive. In fact, given that the I-pipe can be easily integrated with PREEMPT_RT, it should be fairly simple to obtain a single Linux tree that supports both for the PREEMPT_RT approach and the I-pipe/Fusion approach.
IFF a choice must be made between both PREEMPT_RT and the I-pipe/ Fusion, then further tests would need to be carried out, in particular with regards to scheduling latencies. Based on guidance provided by previous discussion, however, such tests may actually reveal that, as in this case, the actual choice depends on a features vs. performance compromise.
Which brings us back to Paul's earlier summary: it appears that there is not a single approach to solve all real-time problems in Linux. Instead, it seems that a clever combination of many approaches is the best way forward. In that regard, we hope that the above tests will help guide this analysis in a fruitful direction.
As usual, your feedback is appreciated.
Best regards,
Kristian Benoit
Karim Yaghmour
--
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || 1-866-677-4546
Quoted 10 lines · Kristian Benoit
For
PREEMPT_RTclearly the results are much better than last time. Indeed it appears that, as Ingo predicted, a combination of the proper configuration options and most recent additions givesPREEMPT_RTimportant gains. In comparison to last week's results all measures are lower: average response time, maximum response time, minimum response time, and standard deviation. This is very good. But that's not all.PREEMPT_RTalso comes down virtually neck-to- neck with the I-pipe (and the previous numbers from Adeos) in terms of maximum interrupt response time. Certainly those backingPREEMPT_RT, and others we hope, will find this quite positive.
You might what to try the overall times numbers with the voluntary preemption instead. That option doesn't convert spinlocks and still uses interrupt threads. I'd be surprised that the spinlocks would contribute to that much overhead. Nevertheless, I'll be curious about those results.
bill
Quoted 5 lines · Bill Huey (hui)
You might what to try the overall times numbers with the voluntary preemption instead. That option doesn't convert spinlocks and still uses interrupt threads. I'd be surprised that the spinlocks would contribute to that much overhead. Nevertheless, I'll be curious about those results.
To tell you the truth, we've spent a considerable amount of time as it is on this and we need to move on to other things. So while we may do a repeat, we think the latest results are good enough to feed the discussion for some time. Also, we've finally release the LRTBF, so we hope you'll try it out yourself and share what you find. We'll continue making LRTBF releases with the contributions we get.
In regards to the performance, of PREEMPT_RT, we've also made available the comparative output of LMbench for the various runs as part of the LRTBF release. Have a look at the bottom of this section:
http://www.opersys.com/lrtbf/#latest-results
I don't know what part of PREEMPT_RT causes this, but looking at some of the numbers from this output one is tempted to conclude that PREEMPT_RT causes a very significant impact on system load. And I don't say this lightly. Have a look for example at the local communication latencies between vanilla and PREEMPT_RT when the target is subject to the HD test. For a pipe, this goes from 9.4us to 21.6. For UDP this goes from 22us to 1070us !!! Even on a system without any load, the numbers are similar. Ouch.
I have no envy of starting a flamefest, but to be fair I must mention that the output for the same runs for I-pipe is much more reasonable. In fact it's much closer to vanilla performance. Again, please no flames ... just read the numbers.
Notice that that part of the test (LMbench) does not require a repeat of our complete setup. A simple LMbench comparison even on an isolated machine should yield similar results.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 2 lines · Karim Yaghmour
To tell you the truth, we've spent a considerable amount of time as it is on this and we need to move on to other things. So while we
Understood.
Quoted 8 lines
I don't know what part of
PREEMPT_RTcauses this, but looking at some of the numbers from this output one is tempted to conclude thatPREEMPT_RTcauses a very significant impact on system load. And I don't say this lightly. Have a look for example at the local communication latencies between vanilla andPREEMPT_RTwhen the target is subject to the HD test. For a pipe, this goes from 9.4us to 21.6. For UDP this goes from 22us to 1070us !!! Even on a system without any load, the numbers are similar. Ouch.
I'm involved in other things now, but I wouldn't be surprised if it was some kind of scheduler bug + softirq wacked interaction. softirqs, from the last time I looked, was pretty raw in RT. Another thing to do is to subtract the number of irq-thread context switches from the total system context switch to see if there's any kind of oddities with the spinlock conversion. I doubt there's going to be a ton of 'overscheduling', nevertheless it would be a valuable number. This is such a new patch that weird things like this are likely, but it's going to take an investigation to see what the real cause is.
FreeBSD went through some slow down when they moved to SMPng, but not the kind of numbers you show for things surrounding the network stack. Something clearly bad happened.
bill
Quoted 3 lines · Bill Huey (hui)
FreeBSD went through some slow down when they moved to SMPng, but not the kind of numbers you show for things surrounding the network stack. Something clearly bad happened.
Note that the numbers are not freak accidents, they are consistent accross the various setups. So in total, that's 15 LMbench runs, all showing consistent _severe_ cost for preempt_rt. And this despite the fact that it comes down neck-to-neck with the ipipe on interrupt response time in those same tests. I would highly suggest setting up an automated benchmark for automatically running LMbench on every preempt_rt release and compare that to the vanilla kernel.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 8 lines · Karim Yaghmour
Quoted 3 lines · Bill Huey (hui)
FreeBSD went through some slow down when they moved to SMPng, but not the kind of numbers you show for things surrounding the network stack. Something clearly bad happened.
Note that the numbers are not freak accidents, they are consistent accross the various setups. So in total, that's 15 LMbench runs, all showing consistent _severe_ cost for
preempt_rt. And this despite
Yeah, I looked at the numbers more carefully and it's clearly showing some problems. There is a significant context switch penalty with irq-threads and I wouldn't surprise if this is what's going on.
bill
Quoted 6 lines · Bill Huey
Quoted 3 lines
target is subject to the HD test. For a pipe, this goes from 9.4us to 21.6. For UDP this goes from 22us to 1070us !!! Even on a system without any load, the numbers are similar. Ouch.
I'm involved in other things now, but I wouldn't be surprised if it was some kind of scheduler bug + softirq wacked interaction. [...]
the UDP-over-localhost latency was a softirq processing bug that is fixed in current PREEMPT_RT patches. (real over-the-network latency was never impacted, that's why it wasnt noticed before.)
Ingo
Quoted 3 lines · Ingo Molnar
the UDP-over-localhost latency was a softirq processing bug that is fixed in current
PREEMPT_RTpatches. (real over-the-network latency was never impacted, that's why it wasnt noticed before.)
That's good to hear, but here are some random stats from the idle run:
Measurements | Vanilla | preemp_rt | ipipe
---------------+-------------+----------------+-------------
fork | 95us | 157us (+65%) | 97us (+2%) open/close | 1.4us | 2.1us (+50%) | 1.4us (~) execve | 355us | 452us (+27%) | 365us (+3%) select 500fd | 12.7us | 27.7us (+118%) | 12.7us (~) mmap | 660us | 2886us (+337%) | 673us (+2%) ... | ... | ... | ...
Here's a under ping flood conditions:
Measurements | Vanilla | preemp_rt | ipipe
---------------+-------------+----------------+-------------
fork | 112us | 223us (+99%) | 121us (+8%) open/close | 1.7us | 3.0us (+76%) | 1.8us (+6%) execve | 421us | 652us (+55%) | 467us (+11%) select 500fd | 14.6us | 38.1us (+161%) | 15.5us (+6%) mmap | 760us | 3936us (+418%) | 819us (+8%) ... | ... | ... | ...
etc. it's like this accross the board.
Unless that fix you mention above takes care of these, there's something seriously wrong going on here.
Are there existing LMbench results posted of PREEMPT_RT which one can go back to for comparison?
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 6 lines · Karim Yaghmour
Quoted 3 lines · Ingo Molnar
the UDP-over-localhost latency was a softirq processing bug that is fixed in current
PREEMPT_RTpatches. (real over-the-network latency was never impacted, that's why it wasnt noticed before.)That's good to hear, but here are some random stats from the idle run:
please retest using recent (i.e. today's) -RT kernels. There were a whole bunch of fixes that could affect these numbers. (But i'm sure you know very well that you cannot expect a fully-preemptible kernel to have zero runtime cost. In that sense, if you want to be fair, you should compare it to the SMP kernel, as total preemptability is a similar technological feat and has very similar parallelism constraints.)
Ingo
Quoted 2 lines · Ingo Molnar
if you want to be fair, you should
compare it to the SMP kernel
Well, if you want to be even more fair, you could hold off on publishing benchmark results that compare an experimental, not fully debugged feature with a mature technology.
Lee
Quoted 3 lines · Lee Revell
Well, if you want to be even more fair, you could hold off on publishing benchmark results that compare an experimental, not fully debugged feature with a mature technology.
Would you have applied similar logic had the results been inverted?
Surely the nature of scientific improvement is somewhere along the lines of experiment, compare, enhance, and retry.
If PREEMPT_RT should not be studied, then what good is it to continue talking about it on the LKML or even to continue posting the patches there?
Surely the goal in doing that is to make it better and more acceptable to the larger crowd. And if that is so, then isn't it to everyone's advantage therefore to make a strong case for its adoption?
Did you really expect that no one was going to start running performance tests on preemp_rt somewhere along the way until its developers gave an "official" ok? Isn't it better to know about such results sooner rather than later?
... I'm sorry, I'm somewhat lost here. I can just guess that you're expressing your dissapointment at the results, and that's something I can understand very well. But shouldn't these results encourage you try even harder? Lest you are telling me that that's as good as it gets ... ?
As a side note about the I-pipe (formerly Adeos), it should be noted that, in as far as I can recall, its latency response and performance impact have not varied a lot since its first introduction over 3 years ago. The mechanism's simplicity makes it unlikely to introduce any sort of significant overhead.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 2 lines · Ingo Molnar
please retest using recent (i.e. today's) -RT kernels. There were a whole bunch of fixes that could affect these numbers.
At this point, we're bound to rerun some of the tests. But there's only so many times that one can claim that such and such test isn't good enough because it doesn't have all the latest bells and whistles. Surely there's more to this overhead than just rudimentary bugfixes.
Sorry, but it's just kind of frustrating to put so much time on something like this and have results offhandidly dismissed just because it isn't the truely bleeding edge. These results are similar to our previous testset, which was on another version of preempt_rt, which we were told then had had a bunch of fixes ...
Like I suggested earlier, there should be an automated test by which each preempt_rt release is lmbenched against vanilla.
Quoted 5 lines
(But i'm sure you
know very well that you cannot expect a fully-preemptible kernel to have zero runtime cost. In that sense, if you want to be fair, you should compare it to the SMP kernel, as total preemptability is a similar technological feat and has very similar parallelism constraints.)
With this line of defense I sense things can get hairy fairly rapidely. So I'll try to tread carefully.
Bare in mind here that what we're trying to find out with such tests is what is the bare minimum cost of the proposed rt enhancements to Linux, and how well do these perform in their rt duties, the most basic of which being interrupt latency.
We understand that none of these approaches have zero cost, and we also understand that not all approaches provide the same mechanisms. However, a critical question must be answered:
What is the least-intrusive change, or sets of changes, that can be applied to the Linux kernel to obtain rt, and what mechanisms can, or cannot, be built on top of it (them)? (the unknown here being that rt is defined differently by different people.)
One answer is what we postulated as being a combination of PREEMPT_RT and the Ipipe, each serving a separate problem space.
Speaking just for myself here:
To be honest, however, I have a very hard time, as a user, to convince myself that I should enable preempt_rt under any but the most dire situations given the results I now have in front of me. Surely there's more of an argument than "this will cost you as much as SMP" for someone deploying UP systems, which apparently is the main target of preempt_rt with things like audio and embedded systems.
I want to believe, but accepting more than >50% overhead over regular system calls requires more than just religion ...
Any automated test, as suggested above, that would show some sort of performance impact decrease over release iterations would be helpful for sure. But that 50%+ is going to have to melt significantly along the way ... for me at least.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 4 lines · Karim Yaghmour
Bare in mind here that what we're trying to find out with such tests is what is the bare minimum cost of the proposed rt enhancements to Linux, and how well do these perform in their rt duties, the most basic of which being interrupt latency.
Agreed. I think it makes more sense to keep comparing it against the UP kernel with preempt disabled, since most embedded devices aren't smp.
Quoted 3 lines
To be honest, however, I have a very hard time, as a user, to convince myself that I should enable
preempt_rtunder any but the most dire situations given the results I now have in front
Same here.
Quoted 8 lines · Karim Yaghmour
Quoted 2 lines · Ingo Molnar
please retest using recent (i.e. today's) -RT kernels. There were a whole bunch of fixes that could affect these numbers.
At this point, we're bound to rerun some of the tests. But there's only so many times that one can claim that such and such test isn't good enough because it doesn't have all the latest bells and whistles. Surely there's more to this overhead than just rudimentary bugfixes.
well, it was your choice to benchmark ADEOS against PREEMPT_RT, right? You posted numbers that showed your project in a favorable light while the PREEMPT_RT numbers were more than 100% off. Your second batch of numbers showed a tie, but we still dont know the true correct PREEMPT_RT irq latency values on your hardware, because your testing still had bugs. So a minimum requirement would be to post accurate numbers - you have started this after all.
this thread showcases one of the many reasons why 'vendor sponsored benchmarking' is such a bad idea. I wont post benchmark numbers comparing PREEMPT_RT to 'other' realtime projects. I'm obviously biased, everyone else sees me as biased, so what's the point? Should i pretend i'm not biased towards the stuff i wrote? That would be hypocritical beyond recognition. I dont benchmark PREEMPT_RT against other projects because i know it perfectly well that it is the best thing since sliced bread ;)
Ingo
Quoted 8 lines · Ingo Molnar
well, it was your choice to benchmark ADEOS against
PREEMPT_RT, right? You posted numbers that showed your project in a favorable light while thePREEMPT_RTnumbers were more than 100% off. Your second batch of numbers showed a tie, but we still dont know the true correctPREEMPT_RTirq latency values on your hardware, because your testing still had bugs. So a minimum requirement would be to post accurate numbers - you have started this after all.
Quoted 8 lines
this thread showcases one of the many reasons why 'vendor sponsored benchmarking' is such a bad idea. I wont post benchmark numbers comparing
PREEMPT_RTto 'other' realtime projects. I'm obviously biased, everyone else sees me as biased, so what's the point? Should i pretend i'm not biased towards the stuff i wrote? That would be hypocritical beyond recognition. I dont benchmarkPREEMPT_RTagainst other projects because i know it perfectly well that it is the best thing since sliced bread ;)
This is an unwarranted personal attack.
Should I simply refrain from conducting tests because 4+ years ago I made a suggestion on how to obtain rt performance in Linux? Heck, I didn't even write a single line of code of it, someone else did.
If I wanted to show "my" project in such a good light, would I have gone back and redone tests, and then published them even if those numbers now showed that the "other" project was as good as "mine"? Would I have even listened to any of your suggestions and gone back and had the tests changed to fit your requirements? Would I still be telling that we're going to further fix the tests based on your feedback?
If I benchmarked adeos and preempt_rt it's simply because these are the two patches on top of vanilla Linux that are actively developed and claim to provide true rt on Linux. Let me ask you a simple question: would a benchmark of adeos against nothing but itself been any relevant?
As for the accuracy of the numbers, they are correct in as far as I'm concerned. They aren't to your liking, that's different. You complained about the way we measured irq latency and we promised to repeat with your suggestions. But in this part of the thread, I was simply asking about the over the top impact as seen in the LMbench results. Do we need to fix LMbench too?
I repeat, the software we used is available for you to download. If we've made a mistake, we'll more than gladly acknowledge it.
As for any hint that we're somehow fixing or showing these results to better "sell" anything, then the scripts, drivers and configs we used are all out there, you're more than welcome to show how evil we are.
FWIW, Opersys has no engineering cycles to spare. This current testset and all related human and hardware costs are actually coming straight out of my personal pocket. I have no client paying for this. And, FWIW, I had absolutely no idea what we were going to find when I started this. Certainly I didn't expect that preempt_rt would be able to do as good as the ipipe in terms of interrupt latency, and that's to your credit.
And if you still disagree, then please go ahead and publish your own test results or have others do so and show how wrong we are. We've given you everything we used on our side. If there's anything else you need, let us know.
After all, showing how much of a fraud we are shouldn't be that difficult, you're a very competent developer. And because of that last reason, I have a hard time holding this ad-hominem attack against you. I am dissapointed though.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 5 lines · Karim Yaghmour
FWIW, Opersys has no engineering cycles to spare. This current testset and all related human and hardware costs are actually coming straight out of my personal pocket. I have no client paying for this. And, FWIW, I had absolutely no idea what we were going to find when I started this. Certainly I didn't expect that
What you found with this is the very limit of your patience :)
Quoted 2 lines
preempt_rtwould be able to do as good as the ipipe in terms of interrupt latency, and that's to your credit.
...
Quoted 4 lines
After all, showing how much of a fraud we are shouldn't be that difficult, you're a very competent developer. And because of that last reason, I have a hard time holding this ad-hominem attack against you. I am dissapointed though.
He's probably confusing you from the real FUDers. I don't see you as a FUDer.
He's just resentful fighting with you over attention from the same batch of strippers at last years OLS. :)
bill
Quoted 2 lines · Bill Huey (hui)
He's probably confusing you from the real FUDers. I don't see you as a FUDer.
Thanks, I appreciate the vote of confidence.
Quoted 2 lines
He's just resentful fighting with you over attention from the same batch of strippers at last years OLS. :)
But I don't want to "fight" Ingo. There would just be no point whatsoever with "fighting" with one the best developers Linux has. I started my involvement in these recent threads with a very clear statement that I was open to being shown wrong in having exclusively championed the nanokernel approach in the past. I set out to show myself wrong with these tests and beside some vague expectations, I truely didn't know what I was going to find. I certainly wouldn't have bet a hot-dog on preempt_rt coming neck-to-neck with the ipipe on interrupt latency ... So yes, in doing so some results I've found aren't that nice. But, hell, I didn't invent those results. They are there for anyone to repdroduce or contradict. I have no monopoly over LMbench, PC hardware, the Linux kernel, or anything else used to get those numbers.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 10 lines · Karim Yaghmour
But I don't want to "fight" Ingo. There would just be no point whatsoever with "fighting" with one the best developers Linux has. I started my involvement in these recent threads with a very clear statement that I was open to being shown wrong in having exclusively championed the nanokernel approach in the past. I set out to show myself wrong with these tests and beside some vague expectations, I truely didn't know what I was going to find. I certainly wouldn't have bet a hot-dog on
preempt_rtcoming neck-to-neck with the ipipe on interrupt latency ... So yes, in doing so some results I've found aren't
Yeah, but so what ? don't freak out and take all of this so seriously. It's not like nanokernels are going to disappear when this patch gets broader acceptance. And who cares if you're wrong ? you ? :) Really, get a grip man. :)
And, of course, DUH, making a kernel fully preemptive makes it (near) real time. These aren't unexpected results.
Quoted 4 lines
that nice. But, hell, I didn't invent those results. They are there for anyone to repdroduce or contradict. I have no monopoly over LMbench, PC hardware, the Linux kernel, or anything else used to get those numbers.
Thanks for the numbers, really. I do expect some kind of performance degradation, but there seems to be triggering some oddities with the patch that aren't consistent with some of our expectations.
Be patient. :)
bill
Quoted 1 line · Bill Huey (hui)
Yeah, but so what ? don't freak out and take all of this so seriously.
Bill, it's not like Opersys is VC-funded or anything of that type. We're a self-sufficient operation and as such we cannot afford to waste any time. So when I dedicate close to 6 weeks of man time to a given project, you bet I'm going to take it seriously.
I have no objection to let things cool off, but it's a lot to ask that I do not take seriously attacks on our ability to produce objective results.
We can leave it at that if you want.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
What sort of hardware would you need for the three boxes to run these tests?
I understand that Karim doesn't have time to keep running these tests, but I'm wondering if others would be able to scrape up surplus hardware and setup the test config, accepting configs/kernels from a couple people to test.
I know that I have a large number of slow (<200MHz) pentiums that are just sitting around at home and could be used for this, but I don't know if they would be considered fast enough for any portions of this test (I have a much smaller number of faster machines that could possibly be used)
even if none of the big-name testing facilities want to do this we should be able to get something setup.
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 4 lines · David Lang
I know that I have a large number of slow (<200MHz) pentiums that are just sitting around at home and could be used for this, but I don't know if they would be considered fast enough for any portions of this test (I have a much smaller number of faster machines that could possibly be used)
I don't think that there should be any limitation on the type of hardware being tested. In fact, I would think that having as diverse a test hardware as possible would be a good thing. Many of the embedded platforms are in fact not that far different from those slow pentiums you have lying around.
My 0.02$,
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 9 lines · Karim Yaghmour
Subject: Re:
PREEMPT_RTvs I-PIPE: the numbers, part 2Quoted 4 lines · David Lang
I know that I have a large number of slow (<200MHz) pentiums that are just sitting around at home and could be used for this, but I don't know if they would be considered fast enough for any portions of this test (I have a much smaller number of faster machines that could possibly be used)
Quoted 5 lines
I don't think that there should be any limitation on the type of hardware being tested. In fact, I would think that having as diverse a test hardware as possible would be a good thing. Many of the embedded platforms are in fact not that far different from those slow pentiums you have lying around.
Ok, I'll dig them out and see about getting them setup.
what pinout do I need to connect the printer ports
I'm thinking that the best approach for this would be to setup a static logger and host and then one (or more) target machines, then we can setup a small website on the host that will allow Ingo (and others) to submit kernels for testing, queue those kernels and then run the tests on each one in turn (and if it runs out of kernels to test it re-tests the last one with a longer run)
how much needs to change in userspace between the various tests? I would assume that between the plain, preempt, and RT kernels no userspace changes are needed, what about the other options?
given the slow speed of these systems it would seem to make more sense to have a full kernel downloaded to them rather then having the local box compile it.
does this sound reasonable?
David Lang
Quoted 9 lines
My 0.02$,
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 1 line · David Lang
what pinout do I need to connect the printer ports
For LRTBF you'll find the pinout in the README of the package.
Quoted 6 lines
I'm thinking that the best approach for this would be to setup a static logger and host and then one (or more) target machines, then we can setup a small website on the host that will allow Ingo (and others) to submit kernels for testing, queue those kernels and then run the tests on each one in turn (and if it runs out of kernels to test it re-tests the last one with a longer run)
Things is you're going to need one logger per target. As for a small website, that sounds good enough. Don't know how feasible it would be but it may be desirable to also have a background task that automatically checks for new releases and conducts the tests automatically.
Quoted 3 lines
how much needs to change in userspace between the various tests? I would assume that between the plain, preempt, and RT kernels no userspace changes are needed, what about the other options?
There are no user-space changes needed, but you may need to install a few things that aren't there (LMbench, LTP, hackbench, etc.)
Quoted 3 lines
given the slow speed of these systems it would seem to make more sense to have a full kernel downloaded to them rather then having the local box compile it.
It's your choice really, but if the tests are to be automated, then local compile shouldn't be a problem since you won't be waiting on it personally.
Quoted 1 line
does this sound reasonable?
For me at least.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 12 lines · Karim Yaghmour
Quoted 1 line · David Lang
what pinout do I need to connect the printer ports
For LRTBF you'll find the pinout in the README of the package.
Quoted 6 lines
I'm thinking that the best approach for this would be to setup a static logger and host and then one (or more) target machines, then we can setup a small website on the host that will allow Ingo (and others) to submit kernels for testing, queue those kernels and then run the tests on each one in turn (and if it runs out of kernels to test it re-tests the last one with a longer run)
Quoted 5 lines
Things is you're going to need one logger per target. As for a small website, that sounds good enough. Don't know how feasible it would be but it may be desirable to also have a background task that automatically checks for new releases and conducts the tests automatically.
the only problem with that would be the need for these low-powered boxes to compile the kernel.
Quoted 8 lines
Quoted 3 lines
how much needs to change in userspace between the various tests? I would assume that between the plain, preempt, and RT kernels no userspace changes are needed, what about the other options?
There are no user-space changes needed, but you may need to install a few things that aren't there (LMbench, LTP, hackbench, etc.)
Quoted 7 lines
Quoted 3 lines
given the slow speed of these systems it would seem to make more sense to have a full kernel downloaded to them rather then having the local box compile it.
It's your choice really, but if the tests are to be automated, then local compile shouldn't be a problem since you won't be waiting on it personally.
that depends on how quickly Ingo releases updates, it would be nice to have a system fast enough to run the tests on each version before the next is released :-)
Quoted 10 lines
Quoted 1 line
does this sound reasonable?
For me at least.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 7 lines · Karim Yaghmour
If I wanted to show "my" project in such a good light, would I have gone back and redone tests, and then published them even if those numbers now showed that the "other" project was as good as "mine"? Would I have even listened to any of your suggestions and gone back and had the tests changed to fit your requirements? Would I still be telling that we're going to further fix the tests based on your feedback?
if anything i wrote offended you i'd like to apologize for it. I feel pretty strongly about the stuff i do, but i always regret 99.9% of the flames in the next morning :) Also, i only realized when reading your reply that you took my "vendor sponsored benchmarking" remark literally (and that's my bad too). I never thought of you as a 'vendor' or having any commercial interest in this benchmarking - it was just a stupid analogy from me. I should have written "supporter driven benchmarking" or so - that would still have been a pretty nice flame ;)
also please consider the other side of the coin. You posted numbers that initially put PREEMPT_RT in a pretty bad light. Those numbers are still being linked to from your website, without any indication to suggest that they are incorrect. Even in your above paragraph you are not talking about flawed numbers, you are talking about 'changing the tests to fit my requirements'. Heck i have no 'requirements' other than to see fair numbers. And if adeos/ipipe happens to beat PREEMPT_RT in a fair irq latency test you wont hear a complaint from me. (you might see a patch quite soon though ;)
And i know what irq latencies to expect from PREEMPT_RT. It takes me 5 minutes to do a 10 million samples irq test using LPPTEST, the histogram takes only 200 bytes on the screen, and the numbers i'm getting differ from your numbers - but obviously i cannot run it on your hardware. The rtc_wakeup and built-in latency-tracer numbers differ too. They could be all wrong though, so i'm curious what your independent method will yield.
your lmbench results look accurate and fair, the slowdown during irq-load is a known and expected effect of IRQ threading. If you flood ping a box and generate context-switches instead of plain interrupts, there will be noticeable overhead. I checked some of the lmbench numbers today on my testbox, and while there's overhead, it's significantly less than the 90% degradation you were seeing. That's why i suggested to you to retest using the current base - but you of course dont 'have to'. There were a number of bugs fixed in the past few dozen iterations of patches that affected various components of lmbench :)
Ingo
Quoted 8 lines · Ingo Molnar
if anything i wrote offended you i'd like to apologize for it. I feel pretty strongly about the stuff i do, but i always regret 99.9% of the flames in the next morning :) Also, i only realized when reading your reply that you took my "vendor sponsored benchmarking" remark literally (and that's my bad too). I never thought of you as a 'vendor' or having any commercial interest in this benchmarking - it was just a stupid analogy from me. I should have written "supporter driven benchmarking" or so - that would still have been a pretty nice flame ;)
Thanks for the clarification, sorry if I jumped to the wrong conclusions regarding what you meant to say.
Quoted 9 lines
also please consider the other side of the coin. You posted numbers that initially put
PREEMPT_RTin a pretty bad light. Those numbers are still being linked to from your website, without any indication to suggest that they are incorrect. Even in your above paragraph you are not talking about flawed numbers, you are talking about 'changing the tests to fit my requirements'. Heck i have no 'requirements' other than to see fair numbers. And if adeos/ipipe happens to beatPREEMPT_RTin a fair irq latency test you wont hear a complaint from me. (you might see a patch quite soon though ;)
OK, please recheck the webpage, I've now added a warning specifically to the effect that the numbers need to be rerun. Hopefully this clears things up.
Quoted 7 lines
And i know what irq latencies to expect from
PREEMPT_RT. It takes me 5 minutes to do a 10 million samples irq test using LPPTEST, the histogram takes only 200 bytes on the screen, and the numbers i'm getting differ from your numbers - but obviously i cannot run it on your hardware. Thertc_wakeupand built-in latency-tracer numbers differ too. They could be all wrong though, so i'm curious what your independent method will yield.
Well the method we're using is certainly not absolute. That's why we're providing the scripts. There's no saying that others (outside yourself and us) will find some other outlandish results. But hopefully the more we study these things, the more consistent we can characterize them.
Quoted 9 lines
your lmbench results look accurate and fair, the slowdown during irq-load is a known and expected effect of IRQ threading. If you flood ping a box and generate context-switches instead of plain interrupts, there will be noticeable overhead. I checked some of the lmbench numbers today on my testbox, and while there's overhead, it's significantly less than the 90% degradation you were seeing. That's why i suggested to you to retest using the current base - but you of course dont 'have to'. There were a number of bugs fixed in the past few dozen iterations of patches that affected various components of lmbench :)
I certainly welcome this. Thanks for partly confirming our results, and pointing out that newever versions are better. Like I said earlier, we're bound to repeat our tests with all that's been suggested now ... So we will redo with the version you mentioned earlier. Hopefully this time we'll get it right.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 3 lines · Karim Yaghmour
OK, please recheck the webpage, I've now added a warning specifically to the effect that the numbers need to be rerun. Hopefully this clears things up.
Are you talking about the first run where you left all those expensive PREEMPT_RT debug options enabled?
IMHO those numbers should be taken down, they're completely meaningless.
Lee
Quoted 4 lines · Lee Revell
Are you talking about the first run where you left all those expensive
PREEMPT_RTdebug options enabled?IMHO those numbers should be taken down, they're completely meaningless.
That's just flamebait. Anyone who's ever read an LKML thread knows better than to just trust the topmost parent. As for those who don't read LKML very often, then the "Latest Results" is the section they'll be most interested in and that specific section starts with with a big fat warning.
You can label the results meaningless if you wish, suit yourself.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 7 lines · Karim Yaghmour
To be honest, however, I have a very hard time, as a user, to convince myself that I should enable
preempt_rtunder any but the most dire situations given the results I now have in front of me. Surely there's more of an argument than "this will cost you as much as SMP" for someone deploying UP systems, which apparently is the main target ofpreempt_rtwith things like audio and embedded systems.
What situation do you consider dire?
I do appreciate the testing that you've done and I hope you do more in the future. Remember that PREEMPT_RT is a fast moving object, we need those kinds of tests every day ..
Daniel
Quoted 3 lines · Daniel Walker
I do appreciate the testing that you've done and I hope you do more in the future. Remember that
PREEMPT_RTis a fast moving object, we need those kinds of tests every day ..
Which is exactly what I was suggesting.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 7 lines · Kristian Benoit
We've finally been able to complete a second round of our tests. Unfortunately, this has taken much longer than we ever anticipated. Without going into too much detail, suffice it to say that we ran into problems with each of our intended test configurations. Fortunately, the results are that much more interesting and, we hope, more accurately illustrate the performance of the approaches being compared.
Again, good stuff!
It looks to me that I-PIPE is an example of a "nested OS", with Linux nested within the I-PIPE functionality. One could take the RTAI-Fusion approach, but this measurement is of I-PIPE rather than RTAI-Fusion, right? (And use of RTAI-Fusion might or might not change these results significantly, just trying to make sure I understand what these specific tests apply to.)
Also, if I understand correctly, the interrupt latency measured is to the Linux kernel running within I-PIPE, rather than to I-PIPE itself. Is this the case, or am I confused?
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
It looks to me that I-PIPE is an example of a "nested OS", with Linux nested within the I-PIPE functionality.
Sorry, the I-pipe is likely in the "none-of-the-above" category. It's actually not much of a category itself. For one thing, it's clearly not an RTOS in any sense of the word.
The I-pipe is just a layer that allows multiple pieces of code to share an interrupt stream in a prioritized fashion. It doesn't schedule anything or provide any sort of abstraction whatsoever. Your piece of code just gets a spot in the pipeline and receives interrupts accordingly. Not much nesting there. It's just a new feature in Linux.
Have a look at the patches and description posted by Philippe last Friday for more detail.
Quoted 5 lines
One could take
the RTAI-Fusion approach, but this measurement is of I-PIPE rather than RTAI-Fusion, right? (And use of RTAI-Fusion might or might not change these results significantly, just trying to make sure I understand what these specific tests apply to.)
That's inconsequential. Whether Fusion is loaded or not doesn't preclude a loaded driver to have a higher priority than Fusion itself and therefore continue receiving interrupt even if Fusion itself has disabled interrupts ...
The loading of Fusion would change nothing to these measurements.
Quoted 3 lines
Also, if I understand correctly, the interrupt latency measured is to the Linux kernel running within I-PIPE, rather than to I-PIPE itself. Is this the case, or am I confused?
What's being measured here is a loadable module that allocates an spot in the ipipe that has higher priority than Linux and puts itself there. Therefore, regardless of what other piece of code in the kernel disables interrupts, that specific driver still has its registered ipipe handler called deterministically ...
Don't know, but from the looks of it we're not transmitting on the same frequency ...
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 16 lines · Karim Yaghmour
Quoted 2 lines · Paul E. McKenney
It looks to me that I-PIPE is an example of a "nested OS", with Linux nested within the I-PIPE functionality.
Sorry, the I-pipe is likely in the "none-of-the-above" category. It's actually not much of a category itself. For one thing, it's clearly not an RTOS in any sense of the word.
The I-pipe is just a layer that allows multiple pieces of code to share an interrupt stream in a prioritized fashion. It doesn't schedule anything or provide any sort of abstraction whatsoever. Your piece of code just gets a spot in the pipeline and receives interrupts accordingly. Not much nesting there. It's just a new feature in Linux.
Quoted 2 lines
Have a look at the patches and description posted by Philippe last Friday for more detail.
It is a bit of an edge case for any of the categories.
Quoted 6 lines
Quoted 5 lines
One could take
the RTAI-Fusion approach, but this measurement is of I-PIPE rather than RTAI-Fusion, right? (And use of RTAI-Fusion might or might not change these results significantly, just trying to make sure I understand what these specific tests apply to.)
Quoted 6 lines
That's inconsequential. Whether Fusion is loaded or not doesn't preclude a loaded driver to have a higher priority than Fusion itself and therefore continue receiving interrupt even if Fusion itself has disabled interrupts ...
The loading of Fusion would change nothing to these measurements.
OK...
Quoted 4 lines
Quoted 3 lines
Also, if I understand correctly, the interrupt latency measured is to the Linux kernel running within I-PIPE, rather than to I-PIPE itself. Is this the case, or am I confused?
Quoted 8 lines
What's being measured here is a loadable module that allocates an spot in the ipipe that has higher priority than Linux and puts itself there. Therefore, regardless of what other piece of code in the kernel disables interrupts, that specific driver still has its registered ipipe handler called deterministically ...
Don't know, but from the looks of it we're not transmitting on the same frequency ...
Probably just my not fully understanding I-PIPE (to say nothing of not fully understanding your test setup!), but I would have expected I-PIPE to be able to get somewhere in the handfuls of microseconds of interrupt latency. Looks like it prevents Linux from ever disabling real interrupts -- my first guess after reading your email was that Linux was disabling real interrupts and keeping I-PIPE from getting there in time.
Thanx, Paul
Quoted 7 lines · Paul E. McKenney
Probably just my not fully understanding I-PIPE (to say nothing of not fully understanding your test setup!), but I would have expected I-PIPE to be able to get somewhere in the handfuls of microseconds of interrupt latency. Looks like it prevents Linux from ever disabling real interrupts -- my first guess after reading your email was that Linux was disabling real interrupts and keeping I-PIPE from getting there in time.
Have a look at the announcement just made by Kristian about the LRTBF. There's a tarball with all the code for the drivers, scripts and configs we used.
Nevertheless, maybe it's worth that I clarify the setup further. Here's what we had:
+----------+
| HOST |
+----------+
|
|
| Ethernet LAN
|
/ \
/ \
/ \
/ \
/ \
/ \
/ \
+--------+ SERIAL +--------+
| LOGGER |----------| TARGET |
+--------+ +--------+The logger sends an interrupt to the target every 1ms. Here's the path travelled by this interrupt (L for logger, T for target):
1- L:adeos-registered handler is called at timer interrupt 2- L:record TSC for transmission
3- L:write out to parallel port
4- T:ipipe-registered handler called to receive interrupt 5- T:write out to parallel port
6- L:adeos-registered handler called to receive interrupt 7- L:record TSC for receipt
The response times obtained include all that goes on from 2 to 7, including all hardware-related delays. The target's true response time is from 3.5 to 5.5 (the .5 being the actual time it takes for the signal to reache the pins on the actual physical parallel port outside the computer.)
The time from 2 to 3.5 includes the execution time for a few instructions (record TSC value to RAM and outb()) and the delay for the hardware to put the value out on the parallel port.
The time from 5.5 to 7 includes an additional copy of adeos' interrupt response time. IOW, in all cases, we're at least recording adeos' interrupt response time at least once. Like we explained in our first posting (and as backed up by the data found in both postings) the adeos-to-adeos setup shows that this delay is bound. In fact, we can safely assume that 2*max_ipipe_delay ~= 55us and that 2*average_ipipe_delay ~= 14us. And therefore:
max_ipipe_delay = 27.5usaverage_ipipe_delay = 7usmax_preempt_delay = 55us - max_ipipe_delay = 27.5us average_preempt_delay = 14 us - average_ipipe_delay = 7us
Presumably the 7us above should fit the "handful" you refer to. At least I hope.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 9 lines · Karim Yaghmour
+----------+
| HOST |
+----------+
|
|
| Ethernet LAN
|
/ \
/ \Quoted 8 lines
/ \
/ \
/ \
/ \
/ \
+--------+ SERIAL +--------+
| LOGGER |----------| TARGET |
+--------+ +--------+Karim mean:
+----------+
| HOST |
+----------+
|
|
| Ethernet LAN
|
/ \
/ \
/ \
/ \
/ \
/ \
/ \
+--------+ PARALLEL +--------+
| LOGGER |----------| TARGET |
+--------+ +--------+Kristian
Quoted 10 lines · Karim Yaghmour
Quoted 7 lines · Paul E. McKenney
Probably just my not fully understanding I-PIPE (to say nothing of not fully understanding your test setup!), but I would have expected I-PIPE to be able to get somewhere in the handfuls of microseconds of interrupt latency. Looks like it prevents Linux from ever disabling real interrupts -- my first guess after reading your email was that Linux was disabling real interrupts and keeping I-PIPE from getting there in time.
Quoted 3 lines
Have a look at the announcement just made by Kristian about the LRTBF. There's a tarball with all the code for the drivers, scripts and configs we used.
I see that now, cool!!! And thank you and Kristian for putting this together!
Quoted 51 lines
Nevertheless, maybe it's worth that I clarify the setup further. Here's what we had:
+----------+ | HOST | +----------+ | | | Ethernet LAN | / \ / \ / \ / \ / \ / \ / \ +--------+ SERIAL +--------+ | LOGGER |----------| TARGET | +--------+ +--------+The logger sends an interrupt to the target every 1ms. Here's the path travelled by this interrupt (L for logger, T for target):
1- L:adeos-registered handler is called at timer interrupt 2- L:record TSC for transmission
3- L:write out to parallel port
4- T:ipipe-registered handler called to receive interrupt 5- T:write out to parallel port
6- L:adeos-registered handler called to receive interrupt 7- L:record TSC for receiptThe response times obtained include all that goes on from 2 to 7, including all hardware-related delays. The target's true response time is from 3.5 to 5.5 (the .5 being the actual time it takes for the signal to reache the pins on the actual physical parallel port outside the computer.)
The time from 2 to 3.5 includes the execution time for a few instructions (record TSC value to RAM and
outb()) and the delay for the hardware to put the value out on the parallel port.The time from 5.5 to 7 includes an additional copy of adeos' interrupt response time. IOW, in all cases, we're at least recording adeos' interrupt response time at least once. Like we explained in our first posting (and as backed up by the data found in both postings) the adeos-to-adeos setup shows that this delay is bound. In fact, we can safely assume that 2*max_ipipe_delay ~= 55us and that 2*average_ipipe_delay ~= 14us. And therefore:
Quoted 7 lines
max_ipipe_delay= 27.5usaverage_ipipe_delay= 7usmax_preempt_delay= 55us -max_ipipe_delay= 27.5usaverage_preempt_delay= 14 us -average_ipipe_delay= 7usPresumably the 7us above should fit the "handful" you refer to. At least I hope.
I have big hands, so 7us could indeed qualify as a "handful".
Any insights as to what leads to the larger maximum delay? Some guesses include worst-case cache-miss patterns and interrupt disabling that I missed in my quick scan of the patch.
If I understand your analysis correctly (hah!!!), your breakdown of the maximum delay assumes that the maximum delays for the logger and the target are correlated. What causes this correlation? My (probably hopelessly naive) assumption would be that there would be no such correlation. In absence of correlation, one might approximate the maximum ipipe delay by subtracting the -average- ipipe delay from the maximum preemption delay, for 55us - 7us = 48us. Is this the case, or am I missing something here?
Of course, in the case of the -average- preemption measurements, dividing by two to get the average ipipe delay makes perfect sense.
Whatever the answer to my maximum-delay question, the same breakdown of the raw latency figures would apply to the CONFIG_PREEMPT_RT case, right?
Thanx, Paul
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines · Karim Yaghmour
Quoted 7 lines · Paul E. McKenney
Probably just my not fully understanding I-PIPE (to say nothing of not fully understanding your test setup!), but I would have expected I-PIPE to be able to get somewhere in the handfuls of microseconds of interrupt latency. Looks like it prevents Linux from ever disabling real interrupts -- my first guess after reading your email was that Linux was disabling real interrupts and keeping I-PIPE from getting there in time.
Quoted 6 lines
Quoted 3 lines
Have a look at the announcement just made by Kristian about the LRTBF. There's a tarball with all the code for the drivers, scripts and configs we used.
I see that now, cool!!! And thank you and Kristian for putting this together!
You're welcome. Thanks to Karim and Opersys that showed me the path !! :)
Quoted 65 lines
Quoted 51 lines
Nevertheless, maybe it's worth that I clarify the setup further. Here's what we had:
+----------+ | HOST | +----------+ | | | Ethernet LAN | / \ / \ / \ / \ / \ / \ / \ +--------+ SERIAL +--------+ | LOGGER |----------| TARGET | +--------+ +--------+The logger sends an interrupt to the target every 1ms. Here's the path travelled by this interrupt (L for logger, T for target):
1- L:adeos-registered handler is called at timer interrupt 2- L:record TSC for transmission
3- L:write out to parallel port
4- T:ipipe-registered handler called to receive interrupt 5- T:write out to parallel port
6- L:adeos-registered handler called to receive interrupt 7- L:record TSC for receiptThe response times obtained include all that goes on from 2 to 7, including all hardware-related delays. The target's true response time is from 3.5 to 5.5 (the .5 being the actual time it takes for the signal to reache the pins on the actual physical parallel port outside the computer.)
The time from 2 to 3.5 includes the execution time for a few instructions (record TSC value to RAM and
outb()) and the delay for the hardware to put the value out on the parallel port.The time from 5.5 to 7 includes an additional copy of adeos' interrupt response time. IOW, in all cases, we're at least recording adeos' interrupt response time at least once. Like we explained in our first posting (and as backed up by the data found in both postings) the adeos-to-adeos setup shows that this delay is bound. In fact, we can safely assume that 2*max_ipipe_delay ~= 55us and that 2*average_ipipe_delay ~= 14us. And therefore:
Quoted 7 lines
max_ipipe_delay= 27.5usaverage_ipipe_delay= 7usmax_preempt_delay= 55us -max_ipipe_delay= 27.5usaverage_preempt_delay= 14 us -average_ipipe_delay= 7usPresumably the 7us above should fit the "handful" you refer to. At least I hope.
I have big hands, so 7us could indeed qualify as a "handful".
Any insights as to what leads to the larger maximum delay? Some guesses include worst-case cache-miss patterns and interrupt disabling that I missed in my quick scan of the patch.
Quoted 8 lines
If I understand your analysis correctly (hah!!!), your breakdown of the maximum delay assumes that the maximum delays for the logger and the target are correlated. What causes this correlation? My (probably hopelessly naive) assumption would be that there would be no such correlation. In absence of correlation, one might approximate the maximum ipipe delay by subtracting the -average- ipipe delay from the maximum preemption delay, for 55us - 7us = 48us. Is this the case, or am I missing something here?
Your analysis is correct, but with 600,000 samples, it is possible that we got 2 peeks (perhaps not maximum), one on the logger and one on the target. So in my point of view, the maximum value is probably somewhere between 55us / 2 and 55us - 7us. And probably closer to 55us / 2.
Quoted 8 lines
Of course, in the case of the -average- preemption measurements, dividing by two to get the average ipipe delay makes perfect sense.
Whatever the answer to my maximum-delay question, the same breakdown of the raw latency figures would apply to the
CONFIG_PREEMPT_RTcase, right?Thanx, Paul
Kristian
Quoted 5 lines · Kristian Benoit
Your analysis is correct, but with 600,000 samples, it is possible that we got 2 peeks (perhaps not maximum), one on the logger and one on the target. So in my point of view, the maximum value is probably somewhere between 55us / 2 and 55us - 7us. And probably closer to 55us / 2.
you could try the LPPTEST kernel driver and testlpp utility i integrated into the -RT patchset. It avoids target-side latencies almost completely. Especially since you had problems with parallel interrupts you should give it a go and compare the results.
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Kristian Benoit
Your analysis is correct, but with 600,000 samples, it is possible that we got 2 peeks (perhaps not maximum), one on the logger and one on the target. So in my point of view, the maximum value is probably somewhere between 55us / 2 and 55us - 7us. And probably closer to 55us / 2.
Quoted 4 lines
you could try the LPPTEST kernel driver and testlpp utility i integrated into the -RT patchset. It avoids target-side latencies almost completely. Especially since you had problems with parallel interrupts you should give it a go and compare the results.
correction: logger-side latencies are avoided.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 4 lines
you could try the LPPTEST kernel driver and testlpp utility i integrated into the -RT patchset. It avoids target-side latencies almost completely. Especially since you had problems with parallel interrupts you should give it a go and compare the results.
correction: logger-side latencies are avoided.
Sorry, I don't see this. I've just looked at lpptest.c and it does practically the same thing LRTBF is doing, have a look for yourself at the code in LRTBF.
In fact lpptest.c is probably running at a higher cost on the logger since it executes a copy_to_user() for every single data point collected. In the case of the LRTBF, we just buffer the results in a preallocated buffer and then read them all at once after the testrun.
Unless I'm missing something, there is nothing done in lpptest that we aren't already doing on either side, logger-side latencies included.
As for the interrupt problems, they were pilot error. They disappeared once the APIC was enabled. That's therefore a non-issue.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 8 lines · Karim Yaghmour
Quoted 6 lines · Ingo Molnar
Quoted 4 lines
you could try the LPPTEST kernel driver and testlpp utility i integrated into the -RT patchset. It avoids target-side latencies almost completely. Especially since you had problems with parallel interrupts you should give it a go and compare the results.
Quoted 5 lines
Quoted 1 line
correction: logger-side latencies are avoided.
Sorry, I don't see this. I've just looked at
lpptest.cand it does practically the same thing LRTBF is doing, have a look for yourself at the code in LRTBF.
you should take another look. The crutial difference is that AFAICS lrtbf is using interrupts on _both_ the logger and the target side. lpptest only uses interrupts on the target side (that is what we are measuring), but uses polling _with all interrupts disabled_ on the logger side. This makes things much more reliable, as it's not some complex mix of two worst-case latencies, but a small constant overhead on the logger side and the worst-case latency on the target side. This also means i can run whatever lpptest version on the logger side, i dont have to worry about its latencies because there are none that are variable.
Quoted 3 lines
In fact
lpptest.cis probably running at a higher cost on the logger since it executes acopy_to_user()for every single data point collected. [...]
logger-side overhead does not matter at all, and the 8 bytes copy is not measured in the overhead. (it is also insignificant.)
Quoted 8 lines
[...] In the case of the LRTBF, we just buffer the results in a preallocated buffer and then read them all at once after the testrun.
Unless I'm missing something, there is nothing done in lpptest that we aren't already doing on either side, logger-side latencies included.
As for the interrupt problems, they were pilot error. They disappeared once the APIC was enabled. That's therefore a non-issue.
well, LPPTEST works just fine with the i8259A PIC too. (which is much more common in embedded setups than IO-APICs)
Ingo
Quoted 10 lines · Ingo Molnar
you should take another look. The crutial difference is that AFAICS lrtbf is using interrupts on _both_ the logger and the target side. lpptest only uses interrupts on the target side (that is what we are measuring), but uses polling
_withall interruptsdisabled_on the logger side. This makes things much more reliable, as it's not some complex mix of two worst-case latencies, but a small constant overhead on the logger side and the worst-case latency on the target side. This also means i can run whatever lpptest version on the logger side, i dont have to worry about its latencies because there are none that are variable.
I see. Granted, this is different. We will redo a limited testset with either lpptest or a modified version of LRTBF that does exactly what you describe. Specifically, we will redo the testrun that is the most painfull to vanilla Linux in terms of interrupt latency, the HD test, for both preempt_rt and ipipe.
However, I have very serious doubts that this will make any difference whatsoever. Granted the numbers will be slightly lowever, but it won't invalidate the conclusions previously obtained and it still won't allow any of us to isolate the exact hardware- specific overhead of interrupt delivery. IOW, I want to make sure that it is clear that we're not doing this because we doubt our results. To the contrary, we're doing it to ensure that any doubts regarding our results are dissipated.
Quoted 2 lines
logger-side overhead does not matter at all, and the 8 bytes copy is not measured in the overhead. (it is also insignificant.)
Maybe, but your user-space application does a printf on every data point it gets ... Not the best that can be. The clean thing to do here is to cumulate the stuff in a buffer and dump it all postmortem.
Quoted 2 lines
well, LPPTEST works just fine with the i8259A PIC too. (which is much more common in embedded setups than IO-APICs)
LRTBF doesn't have a problem with the i8259a, it's the hardware we were using that didn't behave properly under high interrupt load. This is a system-specific problem. I haven't run lpptest on the actual target we used, but I have no reason to believe it wouldn't behave the same types of problems we got with LRTBF. There is no difference on the target-side between LRTBF and lpptest.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 12 lines · Kristian Benoit
Quoted 10 lines · Paul E. McKenney
On Wed, Jun 22, 2005 at 11:31:39AM -0400, Karim Yaghmour wrote:
If I understand your analysis correctly (hah!!!), your breakdown of the maximum delay assumes that the maximum delays for the logger and the target are correlated. What causes this correlation? My (probably hopelessly naive) assumption would be that there would be no such correlation. In absence of correlation, one might approximate the maximum ipipe delay by subtracting the -average- ipipe delay from the maximum preemption delay, for 55us - 7us = 48us. Is this the case, or am I missing something here?
Quoted 4 lines
Your analysis is correct, but with 600,000 samples, it is possible that we got 2 peeks (perhaps not maximum), one on the logger and one on the target. So in my point of view, the maximum value is probably somewhere between 55us / 2 and 55us - 7us. And probably closer to 55us / 2.
Possible, but it could also be a large peak and a small one.
Any way of getting the logger's latency separately? Or the target's?
Thanx, Paul
Quoted 1 line · Paul E. McKenney
Any way of getting the logger's latency separately? Or the target's?
Like I said to Ingo in my other response, we're going to use a technique similar to his lpptest (i.e. disable all interrupts and actively wait for a reponse from the target) on the the logger to settle the issue. I very seriously doubt the results will be any different, but we want to make sure that there are no doubts remaining.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 1 line · Paul E. McKenney
Any way of getting the logger's latency separately? Or the target's?
with lpptest (PREEMPT_RT's built-in parallel-port latency driver) that's possible, as it polls the target with interrupts disabled, eliminating much of the logger-side latencies. The main effect is that it's now only a single worst-case latency that is measured, instead of having to have two worst-cases meet.
Here's a rough calculation to show what the stakes are: if there's a 1:100000 chance to trigger a worst-case irq handling latency, and you have 600000 samples, then with lpptest you'll see an average of 6 events during the measurement. With lrtfb (the one Karim used) the chance to see both of these worst-case latencies on both sides of the measurement is 1:10000000000, and you'd see 0.00006 of them during the measurement. I.e. the chances of seeing the true max latency are pretty slim.
So if you want to reliably measure worst-case latencies in your expected lifetime, you truly never want to serially couple the probabilities of worst-case latencies on the target and the logger side.
Ingo
Quoted 6 lines · Ingo Molnar
with lpptest (PREEMPT_RT's built-in parallel-port latency driver) that's possible, as it polls the target with interrupts disabled, eliminating much of the logger-side latencies. The main effect is that it's now only a single worst-case latency that is measured, instead of having to have two worst-cases meet.
Quoted 7 lines
Here's a rough calculation to show what the stakes are: if there's a 1:100000 chance to trigger a worst-case irq handling latency, and you have 600000 samples, then with lpptest you'll see an average of 6 events during the measurement. With lrtfb (the one Karim used) the chance to see both of these worst-case latencies on both sides of the measurement is 1:10000000000, and you'd see 0.00006 of them during the measurement. I.e. the chances of seeing the true max latency are pretty slim.
If indeed there are 6 events on a single-side which are worst-case, then you would have to also factor in the probability of obtaining an average or below average result on the other side. So again, if all runs were measuring average on each side, one would expect that at least one of the runs would have a bump over the 55us mark. Yet, they all have the same maximum.
Here's an overview of the results spread in the case of IRQ latency measurements in the HD case (this is just a view of one case, a true study would require drawing graphs showing the spread for all tests):
Of 833,000 results for PREEMPT_RT:
- 36 values are above 50us (0.0045% or 4.5/100,000)
- 860 values are 19us and above
Of 781,000 results for IPIPE:
- 213 values are above 50us (0.0273% or 27/100,000)
- 311 values are 19us and above
Contrary to the illustration you make above, it would seem that the fact that both machines are running the same mechanism, the blue-moon effect multiplies upward instead of downward. This, though, is but a preliminary analysis.
Notes:
- Below 19us, the number of measurement points increases for both setups as we get closer to the 14us average mark.
- There are more data points for PREEMPT_RT than ipipe because LMbench takes much more time to complete on the former.
Quoted 3 lines
So if you want to reliably measure worst-case latencies in your expected lifetime, you truly never want to serially couple the probabilities of worst-case latencies on the target and the logger side.
Like I said, we're going to settle this one to avoid any further doubts.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 15 lines · Karim Yaghmour
Quoted 6 lines · Ingo Molnar
with lpptest (PREEMPT_RT's built-in parallel-port latency driver) that's possible, as it polls the target with interrupts disabled, eliminating much of the logger-side latencies. The main effect is that it's now only a single worst-case latency that is measured, instead of having to have two worst-cases meet.
Quoted 7 lines
Here's a rough calculation to show what the stakes are: if there's a 1:100000 chance to trigger a worst-case irq handling latency, and you have 600000 samples, then with lpptest you'll see an average of 6 events during the measurement. With lrtfb (the one Karim used) the chance to see both of these worst-case latencies on both sides of the measurement is 1:10000000000, and you'd see 0.00006 of them during the measurement. I.e. the chances of seeing the true max latency are pretty slim.
Quoted 6 lines
If indeed there are 6 events on a single-side which are worst-case, then you would have to also factor in the probability of obtaining an average or below average result on the other side. So again, if all runs were measuring average on each side, one would expect that at least one of the runs would have a bump over the 55us mark. Yet, they all have the same maximum.
if your likelyhood of getting a 'combo max' event is 1:10000000000 then you'll basically never see the max! What you will see are combinations of lower-order critical paths - i.e. a worst-case path of 35 usecs combined with another, more likely critical path of 20 usecs. You'll still have the statistical appearance of having found a 'max'.
your only hope to have valid results would be if the likelyhood of the maximum path is much higher than the one in my example. But even then, you've significantly reduced the likelyhood of seeing an actual worst-case latency total.
Quoted 1 line
From all the test i've done, 600,000 samples are not enough to trigger
the worst-case latency - even with the polling method! Also, your tests dont really load the system, so you have a fundamentally lower chance of seeing worst-case latencies. My tests do a dd test, a flood ping, an LTP-40-copies test, an rtc_wakeup 8192 Hz test and an infinite loop of hackbench test all in parallel, and even in such circumstances and with a polling approach i need above 1 million samples to hit the worst-case path! (which i cannot know for sure to be the worst-case path, but which i'm reasonably confident about, based on the distribution of the latencies and having done tens of millions of samples in overnight tests.) Obviously it's a much bigger constraint on the IRQ subsystem if _all_ interrupt _and_ DMA sources in the system are as active as possible.
so ... give the -50-12 -RT tree a try and report back the lpptest results you are getting. [ I know the results i am seeing, but i wont post them as a counter-point because i'm obviously biased :-) I'll let people without an axe to grind do the measurements. ]
Ingo
Quoted 13 lines · Ingo Molnar
Quoted 1 line
From all the test i've done, 600,000 samples are not enough to trigger
the worst-case latency - even with the polling method! Also, your tests dont really load the system, so you have a fundamentally lower chance of seeing worst-case latencies. My tests do a dd test, a flood ping, an LTP-40-copies test, an
rtc_wakeup8192 Hz test and an infinite loop of hackbench test all in parallel, and even in such circumstances and with a polling approach i need above 1 million samples to hit the worst-case path! (which i cannot know for sure to be the worst-case path, but which i'm reasonably confident about, based on the distribution of the latencies and having done tens of millions of samples in overnight tests.) Obviously it's a much bigger constraint on the IRQ subsystem if _all_ interrupt _and_ DMA sources in the system are as active as possible.
Hmm... well, I can't say I'm uninterested. Any chances we can get a copy of the scripts you use to do the MOAILT (Mother Of All Irq Latency Tests).
Quoted 2 lines
so ... give the -50-12 -RT tree a try and report back the lpptest results you are getting.
First things first, we want to report back that our setup is validated before we go onto this one. So we've modified LRTBF to do the busy-wait thing.
Quoted 3 lines
[ I know the results i am seeing, but i wont
post them as a counter-point because i'm obviously biased :-) I'll let people without an axe to grind do the measurements. ]
That's an extra reason for giving us a copy (or pointing us to one) of the script you use to run your tests :)
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 3 lines · Karim Yaghmour
Hmm... well, I can't say I'm uninterested. Any chances we can get a copy of the scripts you use to do the MOAILT (Mother Of All Irq Latency Tests).
my 'dohell' script is embarrasingly simple:
while true; do dd if=/dev/zero of=bigfile bs=1024000 count=1024; done & while true; do killall hackbench; sleep 5; done &
while true; do ./hackbench 20; done &
( cd ltp-full-20040707; su mingo -c ./run40; ) &
ping -l 100000 -q -s 10 -f v &
du / &
./dortc &
and i also start a preloaded flood-ping externally. ./dortc does:
chrt -f 98 -p pidof 'IRQ 8'
cd rtc_wakeup
./rtc_wakeup -f 8192 -t 100000
and ./run40 does:
while true; do ./runalltests.sh -x 40; done
it's not rocket science - i'm just starting a sensible mix of latency generators, without letting any of them dominate the landscape.
Quoted 6 lines
Quoted 3 lines
[ I know the results i am seeing, but i wont
post them as a counter-point because i'm obviously biased :-) I'll let people without an axe to grind do the measurements. ]That's an extra reason for giving us a copy (or pointing us to one) of the script you use to run your tests :)
see above. (It's no secret, i described components of this workload in one of my first mails to the adeos thread. Btw., what happened to adeos irq latency testing?)
Ingo
Quoted 2 lines · Ingo Molnar
see above. (It's no secret, i described components of this workload in one of my first mails to the adeos thread.
I remembered your description, but it's always nice to see exactly what's being done. Thanks very much for sending this, we'll integrate it into the LRTBF.
Quoted 2 lines
Btw., what happened to adeos
irq latency testing?)
It got obsoleted by the ipipe testing. It's basically the same thing. What we were testing in the first released testbench was the usage of the interrupt pipeline in adeos. Now that Philippe has forked it out to make it more straightforward for people to look at (instead of thinking they are looking at a true full nanokernel), it was just the appropriate thing to do to use the I-pipe patch instead. The mechansim being measured is exactly the same thing.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 6 lines · Karim Yaghmour
Quoted 2 lines
so ... give the -50-12 -RT tree a try and report back the lpptest results you are getting.
First things first, we want to report back that our setup is validated before we go onto this one. So we've modified LRTBF to do the busy-wait thing.
here's another bug in the way you are testing PREEMPT_RT irq latencies. Right now you are doing this in lrtbf-0.1a/drivers/par-test.c:
if (request_irq ( PAR_TEST_IRQ,
&par_test_irq_handler,
#if CONFIG_PREEMPT_RT
SA_NODELAY,
#else //!CONFIG_PREEMPT_RT
SA_INTERRUPT,
#endif //PREEMPT_RT
you should set the SA_INTERRUPT flag in the PREEMPT_RT case too! I.e. the relevant line above should be:
SA_NODELAY | SA_INTERRUPT,
otherwise par_test_irq_handler will run with interrupts enabled, opening the window for other interrupts to be injected and increasing the worst-case latency! Take a look at drivers/char/lpptest.c how to do this properly. Also, double-check that there is no IRQ 7 thread running on the PREEMPT_RT kernel, to make sure you are measuring irq latencies.
Ingo
Quoted 5 lines · Ingo Molnar
otherwise
par_test_irq_handlerwill run with interrupts enabled, opening the window for other interrupts to be injected and increasing the worst-case latency! Take a look atdrivers/char/lpptest.chow to do this properly. Also, double-check that there is no IRQ 7 thread running on thePREEMPT_RTkernel, to make sure you are measuring irq latencies.
We'll check on this also. Thanks for pointing it out.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 11 lines · Ingo Molnar
Quoted 1 line · Paul E. McKenney
Any way of getting the logger's latency separately? Or the target's?
with lpptest (PREEMPT_RT's built-in parallel-port latency driver) that's possible, as it polls the target with interrupts disabled, eliminating much of the logger-side latencies. The main effect is that it's now only a single worst-case latency that is measured, instead of having to have two worst-cases meet.
Quoted 7 lines
Here's a rough calculation to show what the stakes are: if there's a 1:100000 chance to trigger a worst-case irq handling latency, and you have 600000 samples, then with lpptest you'll see an average of 6 events during the measurement. With lrtfb (the one Karim used) the chance to see both of these worst-case latencies on both sides of the measurement is 1:10000000000, and you'd see 0.00006 of them during the measurement. I.e. the chances of seeing the true max latency are pretty slim.
My concern exactly!
Quoted 3 lines
So if you want to reliably measure worst-case latencies in your expected lifetime, you truly never want to serially couple the probabilities of worst-case latencies on the target and the logger side.
Sounds more practical than the analytical approach! (Take the Laplace transform of the density function, square root, and then take the inverse Laplace transform, if I remember correctly... Which would end up showing a small probability of the maximum latency being the full amount, which ends up really not telling you anything.)
Thanx, Paul
Quoted 5 lines · Kristian Benoit
[...]
Your analysis is correct, but with 600,000 samples, it is possible that we got 2 peeks (perhaps not maximum), one on the logger and one on the target. So in my point of view, the maximum value is probably somewhere between 55us / 2 and 55us - 7us. And probably closer to 55us / 2.
I could provide some help here, by providing the schematics and firmware for having a microcontroller do the pulse timing part. The schematics should be extremely simple, and easy to build in a breadboard (no soldering required) with standard parts from electronics resellers.
With a hardware solution we could measure the *actual* target latency with sub-microsecond accuracy, and do some fun stuff too, like triggering the pulse at random intervals in a given range, etc.
The microcontroller would then connect to the logger (or the HOST in your setup, and avoid an extra computer) through a serial port to report the measurements.
Is this something that could be useful, or do you think this is just overkill?
Signature · 5 lines
Paulo Marques - www.grupopie.com It is a mistake to think you can solve any major problems just with potatoes. Douglas Adams
Quoted 1 line · Paul E. McKenney
I have big hands, so 7us could indeed qualify as a "handful".
:)
Quoted 3 lines
Any insights as to what leads to the larger maximum delay? Some guesses include worst-case cache-miss patterns and interrupt disabling that I missed in my quick scan of the patch.
Beats me. Given that PREEMPT_RT and the I-pipe get to the same maximum by using two entirely different approaches, I'm guessing this has more to do with hardware-related contention than anything inside the patches themselves.
Quoted 3 lines
If I understand your analysis correctly (hah!!!), your breakdown of the maximum delay assumes that the maximum delays for the logger and the target are correlated. What causes this correlation?
No it doesn't. I'm just inferring the maximum and average using the data obtained in the ipipe-to-ipipe setup. In that specific case, I'm assuming that the interrupt latency on both systems for the same type of interrupt is identical (after all, these machines are physically identical, albeit one has 512MB or RAM and the other 256.)
There is no correlation. Just the assumption that what's actually being measured is twice the latency of the ipipe in that specific setup.
Given that the interrupt latency of preempt_rt is measured using one machine runing adeos (read ipipe) and the other preempt_rt, I'm deducing the latency of preempt_rt based on the numbers obtained for the ipipe by looking at the ipipe-to-ipipe setup.
Quoted 5 lines
My (probably hopelessly naive) assumption would be that there would be no such correlation. In absence of correlation, one might approximate the maximum ipipe delay by subtracting the -average- ipipe delay from the maximum preemption delay, for 55us - 7us = 48us. Is this the case, or am I missing something here?
Not directly. You'd have to start by saying that the true maximum ipipe delay is obtained by substracting the average ipipe delay from the measured maximum ipipe delay (to play safe you could even substract the minimum.)
However such a maximum isn't correlated by the data. If indeed there was a difference between the maximums, averages and minimums of the ipipe and preempt_rt, the shear quantity of measurements would not have shown such latency similarities. IOW, it is expected that at least once in a blue moon we'll hit that case where both the target and the logger demonstrate their highest possible latency. That's what we can safely assume 55us is, again given the number of samples. Remember that on the first run, we sometimes observed a maximum ipipe-to-ipipe response time of 21us. That's because in those runs the blue-moon scenario didn't materialize.
Quoted 2 lines
Of course, in the case of the -average- preemption measurements, dividing by two to get the average ipipe delay makes perfect sense.
There's no correlation, so I don't see this one.
Quoted 2 lines
Whatever the answer to my maximum-delay question, the same breakdown of the raw latency figures would apply to the
CONFIG_PREEMPT_RTcase, right?
Sure, but again see the above caveats.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 6 lines · Karim Yaghmour
Quoted 1 line · Paul E. McKenney
I have big hands, so 7us could indeed qualify as a "handful".
:)
Quoted 8 lines
Quoted 3 lines
Any insights as to what leads to the larger maximum delay? Some guesses include worst-case cache-miss patterns and interrupt disabling that I missed in my quick scan of the patch.
Beats me. Given that
PREEMPT_RTand the I-pipe get to the same maximum by using two entirely different approaches, I'm guessing this has more to do with hardware-related contention than anything inside the patches themselves.
Quite possible, perhaps worst-case cache state.
Quoted 31 lines
Quoted 3 lines
If I understand your analysis correctly (hah!!!), your breakdown of the maximum delay assumes that the maximum delays for the logger and the target are correlated. What causes this correlation?
No it doesn't. I'm just inferring the maximum and average using the data obtained in the ipipe-to-ipipe setup. In that specific case, I'm assuming that the interrupt latency on both systems for the same type of interrupt is identical (after all, these machines are physically identical, albeit one has 512MB or RAM and the other 256.)
There is no correlation. Just the assumption that what's actually being measured is twice the latency of the ipipe in that specific setup.
Given that the interrupt latency of
preempt_rtis measured using one machine runing adeos (read ipipe) and the otherpreempt_rt, I'm deducing the latency ofpreempt_rtbased on the numbers obtained for the ipipe by looking at the ipipe-to-ipipe setup.Quoted 5 lines
My (probably hopelessly naive) assumption would be that there would be no such correlation. In absence of correlation, one might approximate the maximum ipipe delay by subtracting the -average- ipipe delay from the maximum preemption delay, for 55us - 7us = 48us. Is this the case, or am I missing something here?
Not directly. You'd have to start by saying that the true maximum ipipe delay is obtained by substracting the average ipipe delay from the measured maximum ipipe delay (to play safe you could even substract the minimum.)
Quoted 10 lines
However such a maximum isn't correlated by the data. If indeed there was a difference between the maximums, averages and minimums of the ipipe and
preempt_rt, the shear quantity of measurements would not have shown such latency similarities. IOW, it is expected that at least once in a blue moon we'll hit that case where both the target and the logger demonstrate their highest possible latency. That's what we can safely assume 55us is, again given the number of samples. Remember that on the first run, we sometimes observed a maximum ipipe-to-ipipe response time of 21us. That's because in those runs the blue-moon scenario didn't materialize.
Quite possible, depending on what the raw distribution of times looks like. If there are a smallish number of 55us events (as there would have to be given an average of 7us), the blue-moon scenario would lead one to expect a much larger number of ~30us events (27.5us + 3.5us).
In absence of a ~30us bulge, there would still be the possibility that one might see an even bluer (violet?) moon that might stack up to ~100us. Heck, there might be that possibility anyway, but such is life when measuring latencies. :-/
(And, yes, there are other CDFs lacking a 30us bulge that would be consistent with a 55us "blue-moon" bulge -- so I guess I am asking if you have the CDF or the raw latency measurements -- though the data set might be a bit large... And I would have to think about how one goes about deriving individual-latency CDF(s) given a single dual-latency CDF, assuming that this is even possible...)
Quoted 4 lines
Quoted 2 lines
Of course, in the case of the -average- preemption measurements, dividing by two to get the average ipipe delay makes perfect sense.
There's no correlation, so I don't see this one.
You are right that there might not be a correlation, and that it might be OK to just divide the maximum latency by two, but I can imagine cases where dividing by two was not appropriate.
Quoted 4 lines
Quoted 2 lines
Whatever the answer to my maximum-delay question, the same breakdown of the raw latency figures would apply to the
CONFIG_PREEMPT_RTcase, right?Sure, but again see the above caveats.
Thanks for the info!
Thanx, Paul
Quoted 6 lines · Paul E. McKenney
(And, yes, there are other CDFs lacking a 30us bulge that would be consistent with a 55us "blue-moon" bulge -- so I guess I am asking if you have the CDF or the raw latency measurements -- though the data set might be a bit large... And I would have to think about how one goes about deriving individual-latency CDF(s) given a single dual-latency CDF, assuming that this is even possible...)
This is a bandwidth issue. The compressed archive containing the interrupt latencies of all our test runs is 100MB. I could provide a URL _privately_ to a handful of individuals, but beyond that someone's going to have to host it.
Let me know if you want this.
Of course, now that LRTBF is out there, others can generate their own data sets.
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 9 lines · Karim Yaghmour
Quoted 6 lines · Paul E. McKenney
(And, yes, there are other CDFs lacking a 30us bulge that would be consistent with a 55us "blue-moon" bulge -- so I guess I am asking if you have the CDF or the raw latency measurements -- though the data set might be a bit large... And I would have to think about how one goes about deriving individual-latency CDF(s) given a single dual-latency CDF, assuming that this is even possible...)
Quoted 6 lines
This is a bandwidth issue. The compressed archive containing the interrupt latencies of all our test runs is 100MB. I could provide a URL _privately_ to a handful of individuals, but beyond that someone's going to have to host it.
Let me know if you want this.
The approach of measuring the target's and the logger's latencies separately is a -much- better approach than using strange mathematical techniques with strange mathematical assumptions. So please don't waste any further time on my misguided request for the full data set!
Quoted 2 lines
Of course, now that LRTBF is out there, others can generate their own data sets.
True enough!
Thanx, Paul
Quoted 4 lines · Karim Yaghmour
max_ipipe_delay= 27.5usaverage_ipipe_delay= 7usmax_preempt_delay= 55us -max_ipipe_delay= 27.5usaverage_preempt_delay= 14 us -average_ipipe_delay= 7us
Ingo, what's the status of putting irq 0 back in a thread with PREEMPT_RT? IIRC this had some adverse (maybe unfixable?) effects so it was disabled a few months ago.
I don't think there's much point in comparing i-pipe to PREEMPT_RT if we know that 21usec pipeline effect from the timer IRQ (see list archives) is still there.
Lee
Quoted 7 lines · Lee Revell
Ingo, what's the status of putting irq 0 back in a thread with
PREEMPT_RT? IIRC this had some adverse (maybe unfixable?) effects so it was disabled a few months ago.I don't think there's much point in comparing i-pipe to
PREEMPT_RTif we know that 21usec pipeline effect from the timer IRQ (see list archives) is still there.
A link to the archives somewhere would be greatly appreciated, this is a very high traffic list after all...
Thanks,
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Re: PREEMPT_RT & threading IRQ 0
In reply to Lee RevellQuoted 3 lines · Lee Revell
Ingo, what's the status of putting irq 0 back in a thread with
PREEMPT_RT? IIRC this had some adverse (maybe unfixable?) effects so it was disabled a few months ago.
the jury is still out on that one - but right now it seems it's too much complexity for a handful of usecs of latency improvement. Especially with things like high-resolution timer support, the threading IRQ0 doesnt seem to be worth it.
Ingo