Skip to contentSkip to conversation

Streamlining code base?

PATCH In review Daniel Wagner 4 participants 10 messages Sep 2, 2020 – Sep 8, 2020

lore
Share & tools
lore
#0 lore raw

cover Streamlining code base?

Hi Clark and John,

I'd like to streamline all tools, so that the usage, command arguments and the output is more consistent. Every tool seems to be a bit different which makes all automatization around the tools more complex than needed.

Eventually, I'd like to add an easy machine parseable output from all tools. So what do you think? Stupid effort, go away?

Here are a bunch of patches to illustrate what I mean. Before invest too much time I'd like to discuss it, e.g. format.

Cheers,
Daniel

Daniel Wagner (4):
cyclicdeadline: Streamline usage output and man page
deadline_test: Streamline usage output and man page
oslat: Streamline usage output and man page
oslat: Use string parser utilies

 src/include/rt-utils.h              |   1 +
 src/lib/rt-utils.c                  |  30 ++++++
 src/oslat/oslat.8                   |   2 +-
 src/oslat/oslat.c                   | 143 ++++++++--------------------
 src/sched_deadline/cyclicdeadline.8 |  24 ++---
 src/sched_deadline/cyclicdeadline.c |  41 ++++----
 src/sched_deadline/deadline_test.8  |  37 ++++---
 src/sched_deadline/deadline_test.c  |  47 +++++----
 8 files changed, 145 insertions(+), 180 deletions(-)
Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/4 cyclicdeadline: Streamline usage output and man page

---
 src/sched_deadline/cyclicdeadline.8 | 24 ++++++++---------
 src/sched_deadline/cyclicdeadline.c | 41 +++++++++++++++--------------
 2 files changed, 33 insertions(+), 32 deletions(-)
src/sched_deadline/cyclicdeadline.8 +12 -12
index 91766fb391c4..6df7faf445f9 100644
@@ -17,7 +17,7 @@ cyclicdeadline \- This program is used to test the deadline scheduler (SCHED_DEA
 .PP
 .SH SYNOPSIS
 .B cyclicdeadline
-.RI "[-ha] [-c cpulist] [-i interval] [-s step] [-t nr_threads] [-D duration]
+.RI "[-ha] [-c CPUSET] [-D TIME] [-i INTV] [-s STEP] [-t NUM]"
 .PP
 .SH DESCRIPTION
 .B cyclicdeadline
@@ -25,28 +25,28 @@ is a cyclictest style program for testing the deadline scheduler
 .PP
 .SH OPTIONS
 .TP
-.B \-h
-Show this help menu
-.TP
 .B \-a
 Use all CPUs
 .TP
-.B \-c cpulist
+.B \-c CPUSET
 Comma / hypen separated list of CPUs to run deadline tasks on
 .TP
-.B \-i interval
+.B \-D TIME
+Specify a length for the test to run
+Append 'm', 'h', or 'd' to specify minutes, hours, or days
+.TP
+.B \-h
+Show this help menu
+.TP
+.B \-i INTV
 The shortest deadline for the tasks in us. (default 1000us)
 .TP
-.B \-s step
+.B \-s STEP
 The amount to increase the deadline for each task in us. (default 500us)
 .TP
-.B \-t nr_threads
+.B \-t NUM
 The number of threads to run as deadline (default 1)
-.TP
-.B \-D time
-Specify a length for the test to run
 .br
-Append 'm', 'h', or 'd' to specify minutes, hours, or days
 .SH AUTHOR
 cyclicdeadline was written by Steven Rostedt <[email protected]>
 .PP
src/sched_deadline/cyclicdeadline.c +21 -20

55 lines not on this page · raw message

Signature · 1 line
2.28.0
#2 lore raw

PATCH 2/4 deadline_test: Streamline usage output and man page

In reply to Daniel Wagner
---
 src/sched_deadline/deadline_test.8 | 37 ++++++++++++-----------
 src/sched_deadline/deadline_test.c | 47 +++++++++++++++---------------
 2 files changed, 41 insertions(+), 43 deletions(-)
src/sched_deadline/deadline_test.8 +18 -19
index 5dc99c0bfc63..8f32c5b6feb6 100644
@@ -19,45 +19,44 @@ deadline_test \- High resolution test program
 This program is used to test the deadline scheduler (SCHED_DEADLINE tasks)
 .SH SYNOPSIS
 .B deadline_test
-.RI "[ \-hb ] [ \-r prio ] [ \-c cpulist ] [ \-i interval ] [ \-p percent ] [ \-P percent ] [ \-t threads ] [ \-s step[us) ]"
-
+.RI "[ \-hb ] [ \-c CPUSET ] [ \-i INTV ] [ \-p PERCENT ] [ \-P PERCENT ] \
+[ \-r PRIO ]  [ \-s STEP ] [ \-t NUM ]"
 .SH OPTIONS
 .TP
-.B \-h
-Show this help menu
-.br
-.TP
 .B \-b
 Bind on the last cpu. (shortcut for -c <lastcpu>)
 .br
 .TP
-.B \-r prio
-Add an RT task with given prio to stress system
+.B \-c CPUSET
+Comma/hyphen separated list of CPUs to run deadline tasks on
 .br
 .TP
-.B \-c cpulist
-Comma/hyphen separated list of CPUs to run deadline tasks on
+.B \-h
+Show this help menu
 .br
 .TP
-.B \-i interval
+.B \-i INTV
 The shortest deadline for the tasks
 .br
 .TP
-.B \-p percent
+.B \-p PERCENT
 The percent of bandwidth to use (1-90%)
 .br
 .TP
-.B \-P percent
-The percent of runtime for execution completion
-              (Default 100%)
+.B \-P PERCENT
+The percent of runtime for execution completion (default 100%)
 .br
 .TP
-.B \-t threads
-The number of threads to run as deadline (default 1)
+.B \-r PRIO
+Add an RT task with given prio to stress system
 .br
 .TP
-.B \-s step(us)
-The amount to increase the deadline for each task (default 500us)
+.B \-s STEP
+The amount to increase the deadline for each task in us (default 500us)
+.br
+.TP
+.B \-t NUM
+The number of threads to run as deadline (default 1)
 .br
 .SH AUTHOR
 Deadline test was written by Steven Rostedt <[email protected]>
src/sched_deadline/deadline_test.c +23 -24

64 lines not on this page · raw message

Signature · 1 line
2.28.0
#3 lore raw

PATCH 3/4 oslat: Streamline usage output and man page

In reply to Daniel Wagner
---
 src/oslat/oslat.8 |  2 +-
 src/oslat/oslat.c | 76 +++++++++++++++++++++++------------------------
 2 files changed, 38 insertions(+), 40 deletions(-)
src/oslat/oslat.8 +1 -1
index 85f2c5bcdf5c..b481d8d82783 100644
@@ -36,7 +36,7 @@ Total memory usage will be this value multiplies 2*N,
 because there will be src/dst buffers for each thread, and
 N is the number of processors for testing.
 .TP
-.B \-t, \-\-runtime=SEC
+.B \-D, \-\-duration=TIME
 Specify test duration, e.g., 60, 20m, 2H (m/M: minutes, h/H: hours, d/D: days).
 By default the unit is s/second.
 .TP
src/oslat/oslat.c +37 -39

108 lines not on this page · raw message

Signature · 1 line
2.28.0
#4 lore raw

PATCH 4/4 oslat: Use string parser utilies

In reply to Daniel Wagner

Use available parse_time_string() instead locally implemented version. While at it move the mem parser helper to the global utility header.

---
 src/include/rt-utils.h |  1 +
 src/lib/rt-utils.c     | 30 +++++++++++++++++++
 src/oslat/oslat.c      | 67 ++----------------------------------------
 3 files changed, 33 insertions(+), 65 deletions(-)
src/include/rt-utils.h +1 -0
index 4ed1cbc53ece..d7f369cdbb47 100644
@@ -28,6 +28,7 @@ uint32_t string_to_policy(const char *str);
 pid_t gettid(void);
 
 int parse_time_string(char *val);
+int parse_mem_string(char *str, uint64_t *val);
 
 void enable_trace_mark(void);
 void tracemark(char *fmt, ...) __attribute__((format(printf, 1, 2)));
src/lib/rt-utils.c +30 -0
index f786588706cd..f9cbfb82febb 100644
@@ -362,6 +362,36 @@ int parse_time_string(char *val)
 	return t;
 }
 
+int parse_mem_string(char *str, uint64_t *val)
+{
+	char *endptr;
+	int v = strtol(str, &endptr, 10);
+
+	if (!*endptr)
+		return v;
+
+	switch (*endptr) {
+	case 'g':
+	case 'G':
+		v *= 1024;
+	case 'm':
+	case 'M':
+		v *= 1024;
+	case 'k':
+	case 'K':
+		v *= 1024;
+	case 'b':
+	case 'B':
+		break;
+	default:
+		return -1;
+	}
+
+	*val = v;
+
+	return 0;
+}
+
 void open_tracemark_fd(void)
 {
 	char path[MAX_PATH];
src/oslat/oslat.c +2 -65

85 lines not on this page · raw message

Signature · 1 line
2.28.0
#5 lore raw
In reply to Daniel Wagner
Quoted 9 lines · On Wed, 2 Sep 2020 10:27:21 +0200 Daniel Wagner <[email protected]> wrote:

Hi Clark and John,

I'd like to streamline all tools, so that the usage, command arguments and the output is more consistent. Every tool seems to be a bit different which makes all automatization around the tools more complex than needed.

Eventually, I'd like to add an easy machine parseable output from all tools. So what do you think? Stupid effort, go away?

I think it's a noble goal and I'd be up for it, especially trying to come up with a parsable output format. Got any thoughts on it? Personally I'd go for (in order of preference):

XML
JSON
Random Gibberish
Any Damned thing
YAML

Clark

Did I mention that I hate YAML?

Quoted 26 lines

Here are a bunch of patches to illustrate what I mean. Before invest too much time I'd like to discuss it, e.g. format.

Cheers,
Daniel

Daniel Wagner (4):
cyclicdeadline: Streamline usage output and man page
deadline_test: Streamline usage output and man page
oslat: Streamline usage output and man page
oslat: Use string parser utilies

 src/include/rt-utils.h              |   1 +
 src/lib/rt-utils.c                  |  30 ++++++
 src/oslat/oslat.8                   |   2 +-
 src/oslat/oslat.c                   | 143 ++++++++--------------------
 src/sched_deadline/cyclicdeadline.8 |  24 ++---
 src/sched_deadline/cyclicdeadline.c |  41 ++++----
 src/sched_deadline/deadline_test.8  |  37 ++++---
 src/sched_deadline/deadline_test.c  |  47 +++++----
 8 files changed, 145 insertions(+), 180 deletions(-)
Signature · 1 line
2.28.0
Signature · 2 lines
The United States Coast Guard
Ruining Natural Selection since 1790
#6 lore raw
Quoted 9 lines · Clark Williams

I think it's a noble goal and I'd be up for it, especially trying to come up with a parsable output format. Got any thoughts on it? Personally I'd go for (in order of preference):

XML
JSON
Random Gibberish
Any Damned thing
YAML

For jitterdebugger I ended up adding a bunch of plugins for the output format. So I don't have to argue which format it best, though if I had to choose I'd properly pick JSON.

Quoted 1 line

Did I mention that I hate YAML?

Named must be your fear before banish it you can.

#7 lore raw
Quoted 11 lines · Daniel Wagner
Quoted 9 lines · Clark Williams

I think it's a noble goal and I'd be up for it, especially trying to come up with a parsable output format. Got any thoughts on it? Personally I'd go for (in order of preference):

XML
JSON
Random Gibberish
Any Damned thing
YAML

Quoted 8 lines

For jitterdebugger I ended up adding a bunch of plugins for the output format. So I don't have to argue which format it best, though if I had to choose I'd properly pick JSON.

Quoted 1 line

Did I mention that I hate YAML?

Named must be your fear before banish it you can.

We have some software called rteval that parses the output of cyclictest.
https://git.kernel.org/pub/scm/utils/rteval/rteval.git/

(I'll take patches for that too)

It runs cyclictest under-the-covers like this
cyclictest -qmu -h 2000 -p95 -t -a
and parses the output.

The output is then transformed into XML.

What I like about this, is that it keeps cyclictest lightweight since the data it parses is just text. Then rteval could generate whatever kind of output you are interested in. xml for clark, yaml for the yammering crowd and so on.

I'm not saying this is the only legitimate approach, but I would like to keep the individual tools lightweight in rt-tests, so that you could run them individually in the embedded space as well as on large machines.

So one way to start would be to make sure that the other tools in the rt-tests suite are capable of generating a similar kind of output, that could then be processed by other tools.

John

#8 lore raw
Quoted 3 lines · John Kacur

I'm not saying this is the only legitimate approach, but I would like to keep the individual tools lightweight in rt-tests, so that you could run them individually in the embedded space as well as on large machines.

Sure, though I still think it would make sense to keep it flexible and have a compile switch if needed. I'd like to avoid additional steps between the measurement and interpretation. Any time we want to add/change a thing, we need to touch more places.

Furthermore, if you look at my implementation for the JSON output of jitterdebugger is really small:

https://github.com/igaw/jitterdebugger/blob/master/jitterdebugger.c#L158

Adding a XML version for this should be in the same size.

If you insist on a single format (XML) that's also okay for me. Python ships all the necessary libraries these days. Though the JSON format is simpler to transform into native data types such as dictionaries and lists, IMO.

Anyway, I don't have any strong feeling and have no real problem with one format and a transform script.

Quoted 3 lines

So one way to start would be to make sure that the other tools in the rt-tests suite are capable of generating a similar kind of output, that could then be processed by other tools.

Yes, I'd like to get consistent output from all tools.

Though I think we should start from the beginning and getting all command line arguments parser, help texts and man pages consistent.

#9 lore raw
Quoted 13 lines · Daniel Wagner
Quoted 3 lines · John Kacur

I'm not saying this is the only legitimate approach, but I would like to keep the individual tools lightweight in rt-tests, so that you could run them individually in the embedded space as well as on large machines.

Sure, though I still think it would make sense to keep it flexible and have a compile switch if needed. I'd like to avoid additional steps between the measurement and interpretation. Any time we want to add/change a thing, we need to touch more places.

Furthermore, if you look at my implementation for the JSON output of jitterdebugger is really small:

Quoted 8 lines

https://github.com/igaw/jitterdebugger/blob/master/jitterdebugger.c#L158

Adding a XML version for this should be in the same size.

If you insist on a single format (XML) that's also okay for me. Python ships all the necessary libraries these days. Though the JSON format is simpler to transform into native data types such as dictionaries and lists, IMO.

If you just read a text stream like rteval does, and store in a native data type, then it is trivial to output any data format.

However, your implementation for your jitterdebugger is truly small, I don't want to stomp on your creativity, so spin off a git branch and go to it. I can be persuaded.

John