collecting xenomai statistics
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like
-s send output of xeno-test to [email protected]
-c <name> if -s, send also kernel config file to [email protected] (May be patch xeno-config to emit also the revision of the svn checkout?)
b) Setup an e-mail account [email protected].
c) Add a archiver which generates daily a gzipped tar file of all messages ever sent to [email protected] (e.g. of its mbox). Make it available somewhere on the internet.
d) Write a converter the raw messages into more suitable representation, eg. a MySQL-DB, a spreadsheet format. Extract the raw message and kernel config and store the publicly accessible on the internet. The DB/spreadsheet will contain pointers (URLs) to the raw message/kernel config.
e) Write viewers which present interesting statistics. E.g. X/HTML pages to present a ordered (by architecture, board, version, etc) view of the available results.
I would like to advance like this:
I am not interested in step a), as it is easy for me just to write a trivial script to sent the xeno-test output to any address. Also adding the revision of svn checkout is not very difficult.
For point b+c) I am lacking the necessary rights, but it easy for me to setup a similar account for my tests here. But I would appreciate, if somebody of the core team would help me to get this work done.
For point d+e) I will (after a week of vacation with my son) invest some work. But I would be very interested in knowing exactly which kind of information the core developers are interested in. How should it be presented? In tabular form? Graphs, which ones? And kind of feedback will be evaluated and integrated.
Best regards
Signature · 1 line
Niklaus Giger
Quoted 18 lines · Niklaus Giger
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like
(quoted diff) +0
-2-s send output of xeno-test to [email protected] -c <name> if -s, send also kernel config file to [email protected](May be patch xeno-config to emit also the revision of the svn checkout?)
b) Setup an e-mail account [email protected].
c) Add a archiver which generates daily a gzipped tar file of all messages ever sent to [email protected] (e.g. of its mbox). Make it available somewhere on the internet.
Quoted 8 lines
d) Write a converter the raw messages into more suitable representation, eg. a MySQL-DB, a spreadsheet format. Extract the raw message and kernel config and store the publicly accessible on the internet. The DB/spreadsheet will contain pointers (URLs) to the raw message/kernel config.
e) Write viewers which present interesting statistics. E.g. X/HTML pages to present a ordered (by architecture, board, version, etc) view of the available results.
Do you know http://issaris.org/rtai (the RTAI LiveCD)? It's no longer maintained but may still contain useful ideas, and Takis (Panagiotis Issaris) will likely be happy to answer questions on details of his approach.
Quoted 11 lines
I would like to advance like this:
I am not interested in step a), as it is easy for me just to write a trivial script to sent the xeno-test output to any address. Also adding the revision of svn checkout is not very difficult.
For point b+c) I am lacking the necessary rights, but it easy for me to setup a similar account for my tests here. But I would appreciate, if somebody of the core team would help me to get this work done.
Quoted 6 lines
For point d+e) I will (after a week of vacation with my son) invest some work. But I would be very interested in knowing exactly which kind of information the core developers are interested in. How should it be presented? In tabular form? Graphs, which ones? And kind of feedback will be evaluated and integrated.
I think starting with what Takis once collected would be more than enough for a first round.
I cannot provide you access to the infrastructure, but I vote for granting it. Great commitment!
Jan
Review
Quoted 10 lines · Niklaus Giger
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like
(quoted diff) +0
-2-s send output of xeno-test to [email protected] -c <name> if -s, send also kernel config file to [email protected]
attached patch adds new -m -M flags for xeno-test, (-s flag is taken, for statistics) former for a fixed addy (to be patched later), latter taking any email as arg.
I didnt add -c <name>, since xeno-test already does something similar; if you build with CONFIG_IKCONFIG_PROC=y, xeno-test greps XENO out of /proc/config.gz (probably needs a few more grep terms, and perhaps a -verbose mode which cats the whole thing.)
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet.
I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN.
So Im not so sure that email is the best way here, but it is conceptually simple.
( back in Nov, I set up a gmail acct, and tried to fetch mail from it with a script. gmail wants TLS security, and didnt let me in, so I punted/shelved this. ) Niklaus' message brought me back on this topic.
So Im considering poaching code from LiveCD that does url-encoding, or just using curl to post to some file-upload url.
How would you do this if you only had busybox ??
Anyway, both email and url-upload are suboptimal wrt spam, latter is also a server support issue.
Quoted 3 lines
(May be patch xeno-config to emit also the revision of the svn checkout?)
perhaps as a 4th number on the version, that way xeno-config can stay as is.
[[email protected] bin]# ./xeno-config --version
2.1.50
Id like instead:
2.1.50.941
This seems better than pokinh around a filesystem, looking for the xenomai svn (which may be on the build-host, not the run-host)
Quoted 3 lines
b) Setup an e-mail account [email protected].
getting messages thru the anti-spam filters is the issue here,
Quoted 2 lines
c) Add a archiver which generates daily a gzipped tar file of all messages ever sent to [email protected] (e.g. of its mbox). Make it available somewhere on the internet.
or a daily/weekly digest/tarball
Quoted 2 lines
d) Write a converter the raw messages into more suitable representation, eg. a MySQL-DB, a spreadsheet format. Extract the raw message and kernel config and store the publicly accessible on the internet. The DB/spreadsheet will contain pointers (URLs) to the raw message/kernel config.
certainly not a bad idea. Will simplify collection/selection of data-sets for various things, esp more complicated selections (with ands, ors, etc).
Quoted 3 lines
e) Write viewers which present interesting statistics. E.g. X/HTML pages to present a ordered (by architecture, board, version, etc) view of the available results.
Ive done minimal dabbling with gnuplot, R, both have possibilities.
With gnuplot, I tried graphing the RTD data, failed cuz theres no time column (couldnt figure out how to create/infer a synthetic 'index' column).
It has some capability to select data out of files using awk,etc subcommands, but for complicated data files like xeno-test outputs (multiple sections, different formats), I think it (the selection, reformating capabilities) might be over-matched.
R can apparently manage complex data-sets, and select data out of them. It sounds tremendously capable (after the learning curve) That said, Ive not grokked its use yet.
PATCH 1/1 xeno-test email addition
Quoted 18 lines · Jim Cromie
Quoted 9 lines · Niklaus Giger
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like -s send output of xeno-test to [email protected]
-c <name> if -s, send also kernel config file to [email protected]attached patch adds new -m -M flags for xeno-test, (-s flag is taken, for statistics) former for a fixed addy (to be patched later), latter taking any email as arg.
I didnt add -c <name>, since xeno-test already does something similar; if you build with
CONFIG_IKCONFIG_PROC=y, xeno-test greps XENO out of/proc/config.gz(probably needs a few more grep terms, and perhaps a -verbose mode which cats the whole thing.)
Quoted 5 lines
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet.
I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN.
So Im not so sure that email is the best way here, but it is conceptually simple.
Oof. Now attached.
xeno-test.in +15 -1
@@ -17,6 +17,8 @@ -L writes to logfile (default "test-`uname -r`") (via script) -N <name> same as -L, but prepend "$name-" (without -L, logname="$name-") prepending allows you to give a full path. + -m sends output file to [email protected] + -M <email> sends output file to given addy # following options are passed thru to latency, klatency -s print statistics of sampled data (default on)
@@ -136,8 +138,11 @@ logprefix= prepost= # command to run pre, and post test (ex ntpq -p) -while getopts 'd:shqT:l:H:B:uLN:w:W:p:' FOO ; do +email='[email protected]' +sendit= +while getopts 'd:shqT:l:H:B:uLN:w:W:p:mM:' FOO ; do + case $FOO in s|h|q) pass="$pass -$FOO" ;;
@@ -166,6 +171,11 @@ p) prepost=$OPTARG loadpass="$loadpass -p '$OPTARG'" ;; + M) + email=$OPTARG + sendit=1 ;; + m) + sendit=1 ;; ?) myusage ;; esac
@@ -179,6 +189,10 @@ # restart inside a script invocation, passing all date=`date +%y%m%d.%H%M%S` script -c "./xeno-test $loadpass $pass $*" "$logprefix$logfile-$date" + if [ $sendit == 1 ]; then + echo "mailing $logprefix$logfile-$date to $email" + mail -s 'xeno-test results' $email < "$logprefix$logfile-$date" + fi else if [ "$altwork" != "" ]; then mkload() { exec $altwork; }
Philippe Gerum
AppliedRe: [Xenomai-core] [patch] xeno-test email addition
Quoted 31 lines · Jim Cromie
Quoted 20 lines · Jim Cromie
Quoted 9 lines · Niklaus Giger
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like -s send output of xeno-test to [email protected]
-c <name> if -s, send also kernel config file to [email protected]attached patch adds new -m -M flags for xeno-test, (-s flag is taken, for statistics) former for a fixed addy (to be patched later), latter taking any email as arg.
I didnt add -c <name>, since xeno-test already does something similar; if you build with
CONFIG_IKCONFIG_PROC=y, xeno-test greps XENO out of/proc/config.gz(probably needs a few more grep terms, and perhaps a -verbose mode which cats the whole thing.)Quoted 5 lines
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet.
I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN.
So Im not so sure that email is the best way here, but it is conceptually simple.
Quoted 2 lines
Oof. Now attached.
Applied, thanks.
Signature · 1 line
Philippe.
Re: [Xenomai-core] [patch] xeno-test email addition
Quoted 34 lines · Philippe Gerum
Quoted 33 lines · Jim Cromie
Quoted 31 lines · Jim Cromie
Quoted 11 lines · Niklaus Giger
Hi
Following a suggestion from Philippe Gerum I propose to collect and prepare like this:
a) Make it easy to collect information
add -s/-c option to xeno-test, help text would look like -s send output of xeno-test to [email protected] -c <name> if -s, send also kernel config file to [email protected]
attached patch adds new -m -M flags for xeno-test, (-s flag is taken, for statistics)
former for a fixed addy (to be patched later), latter taking any email as arg.I didnt add -c <name>, since xeno-test already does something similar; if you build with
CONFIG_IKCONFIG_PROC=y, xeno-test greps XENO out of/proc/config.gz
(probably needs a few more grep terms, and perhaps a -verbose mode which cats the whole thing.)The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet. I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN. So Im not so sure that email is the best way here, but it is conceptually simple.
Quoted 3 lines
Quoted 1 line
Oof. Now attached.
Applied, thanks.
Here is its little brother patch which updates the manpage.
Signature · 1 line
Romain LENGLET
ChangeLog +5 -0
@@ -1,3 +1,8 @@ +2006-04-19 Romain Lenglet <[email protected]> + + * doc/man/xeno-test.man.in: Add documentation for the new -m and -M + options added to the xeno-test script. + 2006-04-18 Jim Cromie <[email protected]> * scripts/xeno-test.in: Add -[Mm] options for automated
doc/man/xeno-test.man.in +14 -3
@@ -9,11 +9,11 @@ .\" Xenomai distribution. .\" .pc -.TH XENO-TEST 1 "2005-10-19" "@PACKAGE_VERSION@" "Xenomai" +.TH XENO-TEST 1 "2006-04-19" "@PACKAGE_VERSION@" "Xenomai" .SH NAME -xeno\-test \- Tests and mesures the performance of a Xenomai installation +xeno\-test \- Tests and measures the performance of a Xenomai installation .SH SYNOPSIS -\fBxeno\-test\fP [\fB\-w\fP \fIworkloads\fP] [\fB\-d\fP \fIdevice\fP] [\fB\-W\fP \fIcommand\fP] [\fB\-p\fP \fIcommand\fP] [\fB\-L\fP [\fB\-N\fP \fIprefix\fP]] [\fB\-s\fP] [\fB\-l\fP \fIsamples\fP] [\fB\-h\fP [\fB\-H\fP \fIcategories\fP] [\fB\-B\fP \fIgranularity\fP]] [\fB\-T\fP \fIseconds\fP [\fB\-q\fP]] [\fB\-\-\fP] [\fIargs\fP] ... +\fBxeno\-test\fP [\fB\-w\fP \fIworkloads\fP] [\fB\-d\fP \fIdevice\fP] [\fB\-W\fP \fIcommand\fP] [\fB\-p\fP \fIcommand\fP] [\fB\-L\fP [\fB\-N\fP \fIprefix\fP]] [\fB\-m\fP | \fB\-M\fP \fIemail\fP] [\fB\-s\fP] [\fB\-l\fP \fIsamples\fP] [\fB\-h\fP [\fB\-H\fP \fIcategories\fP] [\fB\-B\fP \fIgranularity\fP]] [\fB\-T\fP \fIseconds\fP [\fB\-q\fP]] [\fB\-\-\fP] [\fIargs\fP] ... .SH DESCRIPTION \fBxeno\-test\fP measures the performance of Xenomai, by executing Xenomai's \fBlatency\fP and \fBklatency\fP tests while generating a high workload on the system. The default command that is executed to simulate workload (if no alternate command is specified with a \fB-W\fP \fIcommand\fP option) is:
@@ -29,6 +29,8 @@ If an invalid option is specified, \fBxeno\-test\fP prints out an usage help message and exits. +You are strongly encouraged to use the \fB\-m\fP option, to anonymously help the Xenomai team collecting statistics about Xenomai's performance on the widest range of systems. + .SH OPTIONS These following options are specific to \fBxeno\-test\fP: .TP
@@ -55,6 +57,12 @@ \fB\-N\fP \fIprefix\fP Prepends \fIprefix\fP\fB\-\fP to the log file name, hence if the \fB\-L\fP option is also specified the log file name is \fIprefix\fP\fB\-test\-\fP\fIkernel_release\fP. This option is useful to create the log file in a different directory, by specifying a \fIprefix\fP that starts with the relative path to that directory. +.TP +\fB\-m\fP +Sends the results of the tests to the Xenomai team's email address, to help collecting statistics about Xenomai's performance. The email is sent using the system's mail(1) command. +.TP +\fB\-M\fP \fIemail\fP +Sends the results of the tests to the specified email address. The email is sent using the system's mail(1) command. .PP The following options are directly passed to the \fBlatency\fP and \fBklatency\fP test commands executed by \fBxeno\-test\fP. If no such options are specified, the \fB\-s \-T 10 \-q\fP options are passed by default by \fBxeno\-test\fP.
@@ -111,5 +119,8 @@ .BR uname (1) .SH HISTORY .TP +2006-04-19 +Updated to document the new -m and -M options. +.TP 2005-10-19 Written by Romain Lenglet <[email protected]>
Philippe Gerum
AppliedRe: [Xenomai-core] [patch] xeno-test email addition
Quoted 7 lines · Romain Lenglet
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet. I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN. So Im not so sure that email is the best way here, but it is conceptually simple.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Oof. Now attached.
Applied, thanks.
Here is its little brother patch which updates the manpage.
Applied, thanks.
Signature · 1 line
Philippe.
Re: [Xenomai-core] [patch] xeno-test email addition
Quoted 8 lines · Philippe Gerum
Quoted 7 lines · Romain Lenglet
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet. I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN. So Im not so sure that email is the best way here, but it is conceptually simple.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Oof. Now attached.
Applied, thanks.
Doh - small err. if [ $sendit = 1 ] needs quotes around the var, for when its undeffd. will patch soon.
Im looking for other things to add, in part to prevent a changelog entry that says "fixed a dumb tested-new-but-not-old, so bug got out" entry :-/
- change the prewired -m email-addy ? (to [email protected])
- add From: header to get past subscriber-only check.
should this be a prewired addy (forex [email protected])
or the user's name ( if so, which ENVAR should we use ? XENO_USER ?)
- verbose (send whole config atm, perhaps others later)
- run xeno-info, xeno-config
- grep more config-items out of config (for non-verbose mode) latency-killers presumably, PREEMPT, others ?
- NPTL availability (kinda overkill, since its absence when needed is already detected)
Can anyone think of other possibly useful raw data ?
I think Ive already got everything that LiveCD collected, plus some.
thanks
jimc
PS. Is "Engines of Creation" another Joe Satriani album ?
PPS. the dude can *rip* a fretboard. In "real-time"
PPPS. I bet Philippe plays a mean air-guitar ;-)
Re: [Xenomai-core] [patch] xeno-test email addition
Quoted 25 lines · Jim Cromie
Quoted 10 lines · Philippe Gerum
Quoted 8 lines · Romain Lenglet
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet. I suspect that the mail looks like spam, and has been rejected, since my hostname is not a real FQDN. So Im not so sure that email is the best way here, but it is conceptually simple.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Oof. Now attached.
Applied, thanks.
Doh - small err. if [ $sendit = 1 ] needs quotes around the var, for when its undeffd. will patch soon.
Im looking for other things to add, in part to prevent a changelog entry that says "fixed a dumb tested-new-but-not-old, so bug got out" entry :-/
Quoted 8 lines
- change the prewired -m email-addy ? (to [email protected])
- add From: header to get past subscriber-only check.
should this be a prewired addy (forex [email protected]) or the user's name ( if so, which ENVAR should we use ?XENO_USER?)
- verbose (send whole config atm, perhaps others later)
- run xeno-info, xeno-config
- grep more config-items out of config (for non-verbose mode) latency-killers presumably, PREEMPT, others ?
I'd say stuff from the kernel hacking section adding overhead including to primary mode threads, like CONFIG_DEBUG_SPINLOCK, CONFIG_FRAME_POINTER, and friends. And also those Adeos switches: CONFIG_IPIPE_STATS, CONFIG_IPIPE_TRACE.
Quoted 6 lines
- NPTL availability (kinda overkill, since its absence when needed is already detected)
Can anyone think of other possibly useful raw data ?
I think Ive already got everything that LiveCD collected, plus some.
Which should be enough to start something sensible, I guess.
Quoted 7 lines
thanks
jimcPS. Is "Engines of Creation" another Joe Satriani album ?
Yes, released in Y2K.
Quoted 2 lines
PPS. the dude can *rip* a fretboard. In "real-time"
Satch plays faster than I can code, which is ok since he does it better too.
Quoted 2 lines
PPPS. I bet Philippe plays a mean air-guitar ;-)
Nope. Philippe is going to call Vai, Petrucci, Lukather, Van Halen and a few others to the rescue, so that the air shivers while the keyboard suffers.
Signature · 1 line
Philippe.
Quoted 2 lines · Jim Cromie
The -M option works, since I just received an email Id sent earlier, but I also sent one to xenomai-core, and it hasnt shown up yet.
In order to avoid spam, the xenomai-core list probably only accept mail from registered members.
Signature · 1 line
Gilles Chanteperdrix.
Quoted 10 lines · Jim Cromie
perhaps as a 4th number on the version, that way xeno-config can stay as is.
[[email protected] bin]# ./xeno-config --version
2.1.50
Id like instead:
2.1.50.941This seems better than pokinh around a filesystem, looking for the xenomai svn
(which may be on the build-host, not the run-host)
If you want to change the version returned by xeno-config, simply modify config/version and re-run scripts/bootstrap before building Xenomai.
Signature · 1 line
Gilles Chanteperdrix.
Quoted 4 lines · Niklaus Giger
But I would be very interested in knowing exactly which kind of information the core developers are interested in. How should it be presented? In tabular form? Graphs, which ones? And kind of feedback will be evaluated and integrated.
I did some similar work once, so I have some ideas. What I would like to have is, given a configuration (.config) build all versions of Adeos+Xenomai for a given set of compilers. Then, present the result in a tabular form. Each cell would give the number of warnings and errors for the corresponding build, with links to browseable build logs. Maybe the cell could have different colors (green, orange, red) depending on the number of warnings and errors.
What would also be interesting, is to test the "distcheck" target. The interesting result being a comparison between the source directory contents and the generated tarball contents (if distcheck is successful).
Signature · 1 line
Gilles Chanteperdrix.