Skip to contentSkip to conversation

Rt-preempt testing best practices

DISCUSSION Tim Bird 3 participants 4 messages May 16, 2007

lore
Share & tools
lore

Tim Bird

author
#0 lore raw

Hi all,

I'm writing to tell you about a project I'm working on. I'm trying to collect as much information as I can about testing real time performance of Linux.

I've started a wiki page at:
http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
(which is on CELF's wiki)

However, I think this material would be better placed
on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
Does anyone have any problem with me moving the content to there, and continuing my attempt to document this on that site?

Note that there's some duplication of content with other pages on the RT wiki. I'll try to clean this up during the move.

Any comments or suggestions are welcome.

If you have done RT testing, and especially if you have published information I can look at (and link to), please contact me.

Thanks!
-- Tim

P.S. This is based on a discussion we had at the RT BOF at ELC last month.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

#1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines

I'm writing to tell you about a project I'm working on. I'm trying to collect as much information as I can about testing real time performance of Linux.

I've started a wiki page at:
http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
(which is on CELF's wiki)

The tests noted in the LKML post on this page are very problematic, ping - -f is not testing RT at all, it keeps the kernel in a very small active page set thus reducing page related penalties, the while loop using dd
is also not too helpfull as it will de-facto run only in memory and cause
absolutely no disk/mass-storage related interaction (try the same with
mount -o remount,sync / first and it will be devastating ! (limited to ext2/ext3/ufs))

The big problem with RT tests published is that they are all looking at the good case, they are loading the system but assuming successfull operations. The worst cases pup up when you run in the error paths of the kernel - then a trivial application can induce very large jitter in the system (run crashme in the background and rerun the tests...)

Also lmbench can give a statistic view of things (and not even that very precisely in some case i.e. context switch measurements are flawed) so this is not of much help for descision makers which variant to use - it does not help if the average performance is good but the mobile phone or mp3 klicks at 1s intervales "deterministically" - so I guess RT benchmarks need a notion of usage-profile to be of value.

Quoted 5 lines

However, I think this material would be better placed
on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
Does anyone have any problem with me moving the content to there, and continuing my attempt to document this on that site?

Note that this page is dedicated to RT-preempt and will most likely not cover the other variants of RT Linux availabe - so it might well be a good idea to maintain an independant page but cross-link them ?

Quoted 7 lines

Note that there's some duplication of content with other pages on the RT wiki. I'll try to clean this up during the move.

Any comments or suggestions are welcome.

If you have done RT testing, and especially if you have published information I can look at (and link to), please contact me.

There are a number of publications related to both benchmarking and analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were published at the real-time Linux Workshops - should I collect them together and send you those as tar.bz2 or just the links ?

hofrat
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFGSnkHnU7rXZKfY2oRAhAkAJ9OYUionz+BULkA90ezFz7QRcHAvACgofnY nZbizb6oIon3AYr5dSJ0ubk=
=gPtS
-----END PGP SIGNATURE-----

#2 lore raw
Quoted 46 lines · Nicholas Mc Guire
Quoted 8 lines

I'm writing to tell you about a project I'm working on. I'm trying to collect as much information as I can about testing real time performance of Linux.

I've started a wiki page at:
http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
(which is on CELF's wiki)

The tests noted in the LKML post on this page are very problematic, ping -f is not testing RT at all, it keeps the kernel in a very small active page set thus reducing page related penalties, the while loop using dd is also not too helpfull as it will de-facto run only in memory and cause absolutely no disk/mass-storage related interaction (try the same with mount -o remount,sync / first and it will be devastating ! (limited to ext2/ext3/ufs))

The big problem with RT tests published is that they are all looking at the good case, they are loading the system but assuming successfull operations. The worst cases pup up when you run in the error paths of the kernel - then a trivial application can induce very large jitter in the system (run crashme in the background and rerun the tests...)

Also lmbench can give a statistic view of things (and not even that very precisely in some case i.e. context switch measurements are flawed) so this is not of much help for descision makers which variant to use - it does not help if the average performance is good but the mobile phone or mp3 klicks at 1s intervales "deterministically" - so I guess RT benchmarks need a notion of usage-profile to be of value.

Quoted 4 lines

However, I think this material would be better placed
on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
Does anyone have any problem with me moving the content to there, and continuing my attempt to document this on that site?

Note that this page is dedicated to RT-preempt and will most likely not cover the other variants of RT Linux availabe - so it might well be a good idea to maintain an independant page but cross-link them ?

Quoted 4 lines

Note that there's some duplication of content with other pages on the RT wiki. I'll try to clean this up during the move.

Any comments or suggestions are welcome.

Quoted 8 lines
Quoted 2 lines

If you have done RT testing, and especially if you have published information I can look at (and link to), please contact me.

There are a number of publications related to both benchmarking and analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were published at the real-time Linux Workshops - should I collect them together and send you those as tar.bz2 or just the links ?

I would definitely be interested in such material myself. We are actually using the RT patches in real world scenarios on probably 50+ systems (some are development workstations and others are hardware-in-the-loop simulation test systems) and they perform very well. However, it would be nice to have a battery of tests (both correctness and determinism)that could be used to compare future versions.

Quoted 1 line

hofrat

-
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
   kr
author
#3 lore raw
In reply to Nicholas Mc Guire
Quoted 1 line · Nicholas Mc Guire

The tests noted in the LKML post on this page are very problematic...

Thanks, Nicholas, this is very helpful information. I remember you mentioning some of this at ELC, and this is exactly the type of information I want to record.

Quoted 3 lines

Note that this page is dedicated to RT-preempt and will most likely not cover the other variants of RT Linux availabe - so it might well be a good idea to maintain an independant page but cross-link them ?

It claims to cover more than just Rt-preempt, but I was hoping for feedback from someone running the site. I'd prefer that my "Best Practices" doc not be limited to just RT-preempt testing, so if the rt.wiki.kernel.org site is really just RT-preempt maybe it would make sense to put it on the embedded linux wiki instead.

Anyone else care to comment? It's just wiki stuff in any event, so it may not matter that much where it lives, as long as people can get to it easily.

Quoted 4 lines

There are a number of publications related to both benchmarking and analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were published at the real-time Linux Workshops - should I collect them together and send you those as tar.bz2 or just the links ?

A list of links would be great! Thanks,
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================