perf & rasd integration plan

The RAS Deamon (rasd) as available at [1] and the development version at [2], allows to enable kernel tracepoints and outputs the tracepoints fields according to the kernel format.

[1] https://git.kernel.org/cgit/utils/ras/rasd.git/
[2] https://git.linaro.org/people/jean.pihet/rasd.git

rasd currently is a duplicate of the perf tool code, ultimately perf and rasd will use the same common code. The goal is to factor out the common code from perf and place it in small librairies in tools/lib.

Here is the code that rasd currently uses and that should be moved to tools/lib:

- debugfs: already in tools/lib/api/fs
. mount and retrieve path

- evlist: tools/perf/util/evlist.[ch]
. create and init new evlist,
. set cpu and thread maps,
. add events to evlist,
. init and use internal event id,
. alloc and mmap events buffers, manage file descriptors, . enable events,
. read events buffers, parse data,
. unmap and free buffers

- evsel: tools/perf/util/evsel.[ch]
. create and init new tracepoints events,
. init and use internal event id,
. open events, manage fds,
. close and free events

- trace-event: tools/perf/util/trace-event.[ch] and tools/perf/util/trace-event-parse.c
. retrieve and parse events format (using event-parse), . print out events fields

- event-parse: already in tools/lib/traceevent/event-parse.[ch] . retrieve and parse events format,
. parse events format and print out events fields

- trace-seq: already in tools/lib/traceevent/trace-seq.c . format output string for event fields

- events plugins: already in tools/lib/traceevent/event-plugin.c

- util: tools/perf/util/util.[ch]
. files open/read,
. manage events attributes,
. various macros

- test events attributes: tools/perf/tests/attr.c
. test_attr__open()

- thread: tools/perf/util/thread_map.[ch] and
- cpu: tools/perf/util/cpumap.[ch]
. init and manage process maps

- xyarray: tools/perf/util/xyarray.[ch]

- syscall: tools/perf/perf-sys.h

- cgroup: tools/perf/util/cgroup.[ch]

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:

Quoted 64 lines

The RAS Deamon (rasd) as available at [1] and the development version at [2], allows to enable kernel tracepoints and outputs the tracepoints fields according to the kernel format.

[1] https://git.kernel.org/cgit/utils/ras/rasd.git/
[2] https://git.linaro.org/people/jean.pihet/rasd.git

rasd currently is a duplicate of the perf tool code, ultimately perf and rasd will use the same common code. The goal is to factor out the common code from perf and place it in small librairies in tools/lib.

Here is the code that rasd currently uses and that should be moved to tools/lib:

- debugfs: already in tools/lib/api/fs
. mount and retrieve path

- evlist: tools/perf/util/evlist.[ch]
. create and init new evlist,
. set cpu and thread maps,
. add events to evlist,
. init and use internal event id,
. alloc and mmap events buffers, manage file descriptors, . enable events,
. read events buffers, parse data,
. unmap and free buffers

- evsel: tools/perf/util/evsel.[ch]
. create and init new tracepoints events,
. init and use internal event id,
. open events, manage fds,
. close and free events

- trace-event: tools/perf/util/trace-event.[ch] and tools/perf/util/trace-event-parse.c
. retrieve and parse events format (using event-parse), . print out events fields

- event-parse: already in tools/lib/traceevent/event-parse.[ch] . retrieve and parse events format,
. parse events format and print out events fields

- trace-seq: already in tools/lib/traceevent/trace-seq.c . format output string for event fields

- events plugins: already in tools/lib/traceevent/event-plugin.c

- util: tools/perf/util/util.[ch]
. files open/read,
. manage events attributes,
. various macros

- test events attributes: tools/perf/tests/attr.c
. test_attr__open()

- thread: tools/perf/util/thread_map.[ch] and
- cpu: tools/perf/util/cpumap.[ch]
. init and manage process maps

- xyarray: tools/perf/util/xyarray.[ch]

- syscall: tools/perf/perf-sys.h

- cgroup: tools/perf/util/cgroup.[ch]

Quoted 7 lines

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just go on and add tools/perf/util/evlist.o et all to be linked directly, as a first step.

Then, as a second step, we can create a tools/lib/perf/evlist.c having what is currently used by both tools/perf/ and tools/rasd/, i.e. what is proven to be useful for something other than perf.

As the need arises, we go on moving things into tools/lib/perf/evlist.c et all from wherever it appeared first, be it from tools/rasd/, tools/perf/util/evlist.c or anywhere else.

Initial rule being that once it is used by multiple tools living in tools/, then it deserves a place in tools/lib/perf/.

Ditto for other stuff currently living in tools/perf/util/.

- Arnaldo

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

Quoted 4 lines · Borislav Petkov

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

yep, it got accepted, ther schedule is:
Friday, October 17, 2014 from 9:00am – Noon
Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

Quoted 4 lines · Borislav Petkov

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

yep, it got accepted, ther schedule is:
Friday, October 17, 2014 from 9:00am – Noon
Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

Quoted 12 lines · On Sun, 5 Oct 2014 20:24:42 +0200 Jiri Olsa <[email protected]> wrote:

Quoted 4 lines · Borislav Petkov

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

yep, it got accepted, ther schedule is:
Friday, October 17, 2014 from 9:00am – Noon
Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

The SSL cert for that site is comic.

1. Expired over three years ago.

2. Was issued to a different domain.

3. Issued by an un-verifiable CA.

This is a joke, right?

Quoted 4 lines · Borislav Petkov

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

yep, it got accepted, ther schedule is:
Friday, October 17, 2014 from 9:00am – Noon
Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

jirka

Hi Jiri,

Quoted 6 lines · Jiri Olsa

Quoted 4 lines · Borislav Petkov

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting approved, please post the time here so people can come.

Quoted 5 lines

yep, it got accepted, ther schedule is:
Friday, October 17, 2014 from 9:00am – Noon
Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

Great! Are there slides that we can download?

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:

Quoted 7 lines

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just go on and add tools/perf/util/evlist.o et all to be linked directly, as a first step.

Then, as a second step, we can create a tools/lib/perf/evlist.c having what is currently used by both tools/perf/ and tools/rasd/, i.e. what is proven to be useful for something other than perf.

As the need arises, we go on moving things into tools/lib/perf/evlist.c et all from wherever it appeared first, be it from tools/rasd/, tools/perf/util/evlist.c or anywhere else.

Initial rule being that once it is used by multiple tools living in tools/, then it deserves a place in tools/lib/perf/.

Ditto for other stuff currently living in tools/perf/util/.

Quoted 9 lines · Arnaldo Carvalho de Melo

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:

Quoted 7 lines

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Quoted 7 lines

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just go on and add tools/perf/util/evlist.o et all to be linked directly, as a first step.

Then, as a second step, we can create a tools/lib/perf/evlist.c having what is currently used by both tools/perf/ and tools/rasd/, i.e. what is proven to be useful for something other than perf.

It would be good to have tools/lib/perf or so with some base implementation to setup and connect to perf buffers. This is useful for tools not only in tools/. The rasd would be a good reference for this regardless if it is in tools/ or not. I am not sure whether and when rasd will be moved there.

Quoted 7 lines

As the need arises, we go on moving things into tools/lib/perf/evlist.c et all from wherever it appeared first, be it from tools/rasd/, tools/perf/util/evlist.c or anywhere else.

Initial rule being that once it is used by multiple tools living in tools/, then it deserves a place in tools/lib/perf/.

So this wouldn't quite work well as it excludes tools not in tools/.

-Robert

Quoted 2 lines

Ditto for other stuff currently living in tools/perf/util/.

Quoted 12 lines

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:

Quoted 7 lines

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just go on and add tools/perf/util/evlist.o et all to be linked directly, as a first step.

Quoted 3 lines

Then, as a second step, we can create a tools/lib/perf/evlist.c having what is currently used by both tools/perf/ and tools/rasd/, i.e. what is proven to be useful for something other than perf.

It would be good to have tools/lib/perf or so with some base implementation to setup and connect to perf buffers. This is useful for tools not only in tools/. The rasd would be a good reference for

this regardless if it is in tools/ or not. I am not sure whether and when rasd will be moved there.

Agreed, we can use rasd as a reference, I'm now looking for the git repo url to see what it is using to prototype how I think this should be done.

Quoted 3 lines

Agreed, we can use rasd as a reference, I'm now looking for the git repo url to see what it is using to prototype how I think this should be done.

https://git.kernel.org/cgit/utils/ras/rasd.git

Em Mon, Oct 06, 2014 at 05:01:47PM +0200, Borislav Petkov escreveu:

Quoted 6 lines · Arnaldo Carvalho de Melo

Quoted 3 lines

Agreed, we can use rasd as a reference, I'm now looking for the git repo url to see what it is using to prototype how I think this should be done.

https://git.kernel.org/cgit/utils/ras/rasd.git

I got Jean's "development" repo, should I get the g.k.o one?

Em Mon, Oct 06, 2014 at 11:07:26AM +0200, Robert Richter escreveu:

Quoted 17 lines · Arnaldo Carvalho de Melo

Quoted 16 lines

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:

Quoted 7 lines

The plan is to move the small and generic functions first: util, xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event, trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just go on and add tools/perf/util/evlist.o et all to be linked directly, as a first step.

Then, as a second step, we can create a tools/lib/perf/evlist.c having what is currently used by both tools/perf/ and tools/rasd/, i.e. what is proven to be useful for something other than perf.

Quoted 3 lines

It would be good to have tools/lib/perf or so with some base implementation to setup and connect to perf buffers. This is useful for tools not only in tools/. The rasd would be a good reference for

Agreed, we can use rasd as a reference, I'm now looking for the git repo url to see what it is using to prototype how I think this should be done.

- Arnaldo

Quoted 2 lines

this regardless if it is in tools/ or not. I am not sure whether and when rasd will be moved there.

- Arnaldo

The RAS Deamon (rasd) as available at [1] and the development version at [2], allows to enable kernel tracepoints and outputs the tracepoints fields according to the kernel format.

[1] https://git.kernel.org/cgit/utils/ras/rasd.git/
[2] https://git.linaro.org/people/jean.pihet/rasd.git

rasd currently is a duplicate of the perf tool code, ultimately perf and rasd will use the same common code. The goal is to factor out the common code from perf and place it in small librairies in tools/lib.

Here is the code that rasd currently uses and that should be moved to tools/lib:

- debugfs: already in tools/lib/api/fs
. mount and retrieve path

- evlist: tools/perf/util/evlist.[ch]
. create and init new evlist,
. set cpu and thread maps,
. add events to evlist,
. init and use internal event id,
. alloc and mmap events buffers, manage file descriptors, . enable events,
. read events buffers, parse data,
. unmap and free buffers

- evsel: tools/perf/util/evsel.[ch]
. create and init new tracepoints events,
. init and use internal event id,
. open events, manage fds,
. close and free events

So, why have you commented out the perf_missing_features fallbacks? Are they getting in the way somehow, what can we do upstream, i.e. in tools/perf/util/ so that it gets closer to what you want to have?

Ditto for all the other ifdef'ed code, can you elaborate on why each is needed? That will help in moving what is used by rasd from tools/perf/util/ to tools/perf/lib/ so that we can eliminate the duplicity.

Quoted 4 lines

So, why have you commented out the perf_missing_features fallbacks? Are they getting in the way somehow, what can we do upstream, i.e. in tools/perf/util/ so that it gets closer to what you want to have?

Quoted 4 lines

Ditto for all the other ifdef'ed code, can you elaborate on why each is needed? That will help in moving what is used by rasd from tools/perf/util/ to tools/perf/lib/ so that we can eliminate the duplicity.

Oh, the reason is very simple: so that it builds. We don't need those facilities yet. We will enable them when needed but we wanted to concentrate on rasd first and later clean up stuff as we go.

Right, and it is being a great exercise, thanks for the patience so far ;-)

Looking at those ifdefs we see things that are specific to tools/perf/, like perf_evsel having a struct hists embedded... I.e. that is of no interest whatsoever (so far) to rasd, and in turn pulls other object files.

So I think that right now we need to look at those ifdefs and go on making what is in tools/perf/util/ stop using it somehow, so that what then gets moved to tools/lib/api/perf/ (I guess this is where it should be, opinions?) have that sorted out.

I.e. what goes to tools/lib/api/perf/ is what is common to the needs of tools/perf/ and rasd (wherever it may roam).

Perhaps something like I did for sock/inet_sock/inet_connection_sock ages ago... I.e. the tools that want a hists will have a

struct hists_evsel {
	struct perf_evsel evsel;
	struct hists	  hists;
};

etc.

Experimenting with that, as it is the only thing ifdefed out in rasd's copy of evsel.h...

Quoted 2 lines

Right, and it is being a great exercise, thanks for the patience so far ;-)

I know, this is your secret agenda to keep me at bay working on this! :-P

Quoted 22 lines

Looking at those ifdefs we see things that are specific to tools/perf/, like perf_evsel having a struct hists embedded... I.e. that is of no interest whatsoever (so far) to rasd, and in turn pulls other object files.

So I think that right now we need to look at those ifdefs and go on making what is in tools/perf/util/ stop using it somehow, so that what then gets moved to tools/lib/api/perf/ (I guess this is where it should be, opinions?) have that sorted out.

I.e. what goes to tools/lib/api/perf/ is what is common to the needs of tools/perf/ and rasd (wherever it may roam).

Perhaps something like I did for sock/inet_sock/inet_connection_sock ages ago... I.e. the tools that want a hists will have a

struct hists_evsel {
	struct perf_evsel evsel;
	struct hists	  hists;
};

etc.

Quoted 2 lines

Experimenting with that, as it is the only thing ifdefed out in rasd's copy of evsel.h...

Right, this all reads like it is going in the right direction but the more important question IMO is are we doing a libperf or are we still doing tools/lib/api/perf/ of single object files to which people can link?

Because if it is the second - and it sounds to me like it is - how do you propose we link with external projects? IOW, if I want to have rasd build *without* the kernel tree, is a simple

cp -r tools/lib/api/ <my_project>/path/to/perf/lib/

work?

I mean, I don't know and I'm just asking. Is that the proposed way? Are we fine with that? Do we want something different, maybe even a lib? Is it time for a lib even?

Because the whole perf functionality is being cravingly ogled by other users - I know Matt wants it for cache QoS or whatever it is called and there are probably a whole lot of projects which would like to process events programatically in userspace.

So what I'm saying is, we probably should have a nice clean way to be able to share that code with external projects instead of external projects duplicating and building a whole library around perf_event_open() and the likes...

Just a couple of thoughts...

Signature · 5 lines

Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

My preference would be for single object files, but the pressure to have a written in stone library seems to just build up...

After doing a 'make perf-targz-src-pkg'

I.e. no kernel sources involved on the machines where I build test.

IOW, it is untangled from the kernel sources. As tools/lib/api/ should as well.

Well, the rasd experience is serving to show areas where there is unnecessary entanglement (hists inside perf_event, etc, the ifdefs you put in place).

I'm working to remove the ones that are in rasd.c, aiming to have a tools/lib/api/ tree that can be used to build rasd and tools/perf/.

What I don't want to do is to simply straight more tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work is needed.

Quoted 2 lines

My preference would be for single object files, but the pressure to have a written in stone library seems to just build up...

Yeah, I think if we have an alternative way to easily integrate that functionality into external projects, we can avoid the need for a lib.

...

Quoted 6 lines

After doing a 'make perf-targz-src-pkg'

I.e. no kernel sources involved on the machines where I build test.

IOW, it is untangled from the kernel sources. As tools/lib/api/ should as well.

Ok, that actually sounds like something we should try:

$ make perf-api-src-pkg

which would spit a tgz with all the generic stuff perf carries. People would simply unpack it in their projects and build it.

That would require somewhat stable function names so that updating is relatively painless and if visible functions change, some sort of a

README file should say what and why.

Sounds like a good alternative to a lib to me...

Quoted 3 lines

Well, the rasd experience is serving to show areas where there is unnecessary entanglement (hists inside perf_event, etc, the ifdefs you put in place).

Yep.

Quoted 4 lines

I'm working to remove the ones that are in rasd.c, aiming to have a tools/lib/api/ tree that can be used to build rasd and tools/perf/.

What I don't want to do is to simply straight more

Quoted 2 lines

tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work is needed.

Yep.

Right, we now should always think that the great advantage of having this code and its users in the same repo tree is going away, i.e. we will not be able to improve the code by fixing mistakes that require changing its users.

Quoted 4 lines

Right, we now should always think that the great advantage of having this code and its users in the same repo tree is going away, i.e. we will not be able to improve the code by fixing mistakes that require changing its users.

Why? I mean, in the rasd case, changing function calling conventions as part of updating the perf libraries is no act. I mean, we won't link against a shared lib but build a static executable and thus the perf src lib (let's call it that - a source library) we're using is a don't care.

So you say that if we change a method, say, perf_evlist__open() to accept a new argument, or change the type of one of its arguments, with a good reason, it is acceptable and with just a change in the README explaining why the build fails, which helps in finding how to fix it in a particular project using these "source libraries" then it would be ok?

That would be fine with me.

We should take care to make sure that the build _fails_ in such cases, i.e.avoid changing the semantic of the Nth argument but keeping its type kind of changes.

At some point there will be no more reasons to change things, and that will be noticed by how long since the last change was made to a particular class, at that point we may well think about making library type promises.

Quoted 6 lines

So you say that if we change a method, say, perf_evlist__open() to accept a new argument, or change the type of one of its arguments, with a good reason, it is acceptable and with just a change in the README explaining why the build fails, which helps in finding how to fix it in a particular project using these "source libraries" then it would be ok?

Right, exactly.

Updating the perf bits used in another project will require a recompilation anyway so adjusting stuff here and there should not be of issue.

Quoted 1 line

That would be fine with me.

Yeah.

Quoted 3 lines

We should take care to make sure that the build _fails_ in such cases, i.e.avoid changing the semantic of the Nth argument but keeping its type kind of changes.

Yep.

Quoted 4 lines

At some point there will be no more reasons to change things, and that will be noticed by how long since the last change was made to a particular class, at that point we may well think about making library type promises.

I was about to say the same thing - once the code doesn't change anymore and gets even boring, doing the library promise is probably pretty easy.

See, no need for you to get over to LCE: we're practically done :-)

We'll run the proposal in jolsa's timeslot again and make sure to poke holes in it. But it looks good from where I'm standing.

Btw, did we just solve the compatibility problem of libraries by delivering source libraries first!?!

Haha.

From 8dc34bd2824c7843182f4fc6deabaf573e42e806 Mon Sep 17 00:00:00 2001

Heya, please check if this is OK.

This was while looking the set of methods used by rasd, trying to reduce it to the bare minimum.

Perhaps even that cpu_map__new() one can be ditched, leaving to use the default of a NULL cpumap that will end up being one with -1, i.e. all cpus.

- Arnaldo

From 8dc34bd2824c7843182f4fc6deabaf573e42e806 Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Mon, 6 Oct 2014 15:43:42 -0300
Subject: [PATCH] rasd: Use perf_evlist__open() instead of open coded
equivalent

Cc Borislav Petkov <[email protected]>

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

---
 src/rasd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

src/rasd.c +0 -1

index 06ccdcdd9d56..fb33fca131fa 100644
@@ -241,7 +241,6 @@ static void daemonize(void)
 
 int main()
 {
-       struct perf_evsel *c;
        struct thread_map *threads;
        struct cpu_map *cpus;
        int i;

@@ -278,12 +277,9 @@ int main()
        perf_evlist__set_maps(evlist, cpus, threads);
 

src/rasd.c +0 -1

        /* mmap buffers */
        if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)
--

1.9.3

src/rasd.c int main() · line 282

Quoted 10 lines

--- a/src/rasd.c

src/rasd.c +3 -6

-       evlist__for_each(evlist, c) {
-               /* On all online cpus by default, system wide tracing */
-               if (perf_evsel__open(c, evlist->cpus, NULL) < 0)
-                       err("opening tracepoint, are you root?");
-       }
-       perf_evlist__set_id_pos(evlist);
+       /* On all online cpus by default, system wide tracing */
+       if (perf_evlist__open(evlist) < 0)
+               err("opening tracepoint, are you root?");

That way the system wide tracing does not work. perf_evlist__open uses a non-NULL thread mapping and so it only traces the events generated by the daemon itself.

perf top uses a similar approach but uses a lot of code in machine.c (machine__synthesize_threads) and util/event.c (perf_event__synthesize_threads) to synthesize the threads etc., which we want to avoid in the minimalistic approach of rasd.

Maybe I missed something about the system wide tracing, any suggestion is welcome.

Thanks for looking!

Jean

Quoted 6 lines

        /* mmap buffers */
        if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)

--
1.9.3

Hopefully we will completely remove the need to set up any thread or cpu map, as what you want is syswide tracing, right?

Quoted 2 lines

Hopefully we will completely remove the need to set up any thread or cpu map, as what you want is syswide tracing, right?

Yeah, I was questioning the need to open at least one thread even for system-wide tracing.

See, source libraries are the new great thing for collaborative development - f*ck "cloud"!

:-D

Quoted 2 lines

Hopefully we will completely remove the need to set up any thread or cpu map, as what you want is syswide tracing, right?

Yeah, I was questioning the need to open at least one thread even for system-wide tracing.

Ok, so what is now at my perf/hists branch at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git

should do the trick, the patch below makes rasd.c use it.

I'll try lding rasd.c with it and checking if it works.

Never having tried this, what are the requisites to test it? Some specific hardware and a kernel with the right tracepoint? I guess some recent 3.17-rc kernel is all that I need?

Quoted 5 lines

I'll try lding rasd.c with it and checking if it works.

Never having tried this, what are the requisites to test it? Some specific hardware and a kernel with the right tracepoint? I guess some recent 3.17-rc kernel is all that I need?

Well, you'd need the part of Jean's patches which adds the event to evlist:

https://lkml.kernel.org/r/[email protected]

AFAICT, you could apply patches 1-5 and replace 6 with yours. Now, rasd.cfg has the mce:mce_record tracepoint which rasd opens but you probably want to put a tracepoint which is much easier to exercise, maybe some syscall or whatever.

I think that should do it but we won't know until we've tried it.

HTH and thanks a lot for doing this!

Signature · 5 lines

Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Right, stoopid me, no need for some specific tracepoint, just to see that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll try that later.

Next stuff I probably will do is to move the bare minimum used by rasd to tools/lib/api/perf/, i.e. there will be:

tools/lib/api/perf/evsel.c
tools/perf/util/evsel.c

Both will share the perf_evsel__ namespace (which I thought at some point to make just: evsel__<METHOD_NAME>, wdyt?).

That way we just make public the bare minimum that already proved to be useful outside tools/perf/ and over time we move stuff from tools/perf/util/evsel.c (and from other tools in or out perf's repo) into the lib.

Np, had to be done at some point :)

Quoted 4 lines · Arnaldo Carvalho de Melo

Right, stoopid me, no need for some specific tracepoint, just to see that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll try that later.

Quoted 8 lines

Next stuff I probably will do is to move the bare minimum used by rasd to tools/lib/api/perf/, i.e. there will be:

tools/lib/api/perf/evsel.c
tools/perf/util/evsel.c

Both will share the perf_evsel__ namespace (which I thought at some point to make just: evsel__<METHOD_NAME>, wdyt?).

Quoted 5 lines

That way we just make public the bare minimum that already proved to be useful outside tools/perf/ and over time we move stuff from tools/perf/util/evsel.c (and from other tools in or out perf's repo) into the lib.

Abso-f*cking-lutely!

:-D

Quoted 1 line

Np, had to be done at some point :)

Yeah, cool. Let me know what I should do to help out.

Thanks.

--
Regards/Gruss,
Boris.

Sent from a fat crate under my desk. Formatting is fine. --

Hi Arnaldo, Borislav,

Quoted 5 lines · Borislav Petkov

Quoted 4 lines · Arnaldo Carvalho de Melo

Right, stoopid me, no need for some specific tracepoint, just to see that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll try that later.

Quoted 8 lines

Quoted 8 lines

Next stuff I probably will do is to move the bare minimum used by rasd to tools/lib/api/perf/, i.e. there will be:

tools/lib/api/perf/evsel.c
tools/perf/util/evsel.c

Both will share the perf_evsel__ namespace (which I thought at some point to make just: evsel__<METHOD_NAME>, wdyt?).

Makes perfectly sense

Quoted 1 line

Quoted 1 line

Quoted 4 lines

Quoted 4 lines

That way we just make public the bare minimum that already proved to be useful outside tools/perf/ and over time we move stuff from tools/perf/util/evsel.c (and from other tools in or out perf's repo) into the lib.

That is great!

About rasd usage, there are some details at
https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.

Quoted 10 lines · Borislav Petkov

Quoted 5 lines · Arnaldo Carvalho de Melo

Quoted 4 lines

That way we just make public the bare minimum that already proved to be useful outside tools/perf/ and over time we move stuff from tools/perf/util/evsel.c (and from other tools in or out perf's repo) into the lib.

That is great!

About rasd usage, there are some details at
https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.

sorry for late reply.. I was on vacation last week..

I read the rasd sources and realized we could poke this from another angle.. AFAIU the work the rasd does is following:
- reads config file and opens configured tracepoints
- reads samples comming from those tracepoints and displays/writes this data to the console/file
- is there more?

If I'm not missing anything, this is quite usefull/common usage pattern which would deserve new perf command.

I can see the analogy with ftrace debugfs interface
- choose/enable tracepoints
- cat .../tracing/trace-pipe

and there could be '-d' for the command to act as daemon.

Em Tue, Oct 14, 2014 at 03:56:31PM +0200, Jiri Olsa escreveu:

Quoted 21 lines · Jean Pihet

Quoted 10 lines · Borislav Petkov

Quoted 5 lines · Arnaldo Carvalho de Melo

Quoted 4 lines

That way we just make public the bare minimum that already proved to be useful outside tools/perf/ and over time we move stuff from tools/perf/util/evsel.c (and from other tools in or out perf's repo) into the lib.

That is great!

About rasd usage, there are some details at
https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.

sorry for late reply.. I was on vacation last week..

I read the rasd sources and realized we could poke this from another angle.. AFAIU the work the rasd does is following:
- reads config file and opens configured tracepoints
- reads samples comming from those tracepoints and displays/writes this data to the console/file
- is there more?

Quoted 8 lines

If I'm not missing anything, this is quite usefull/common usage pattern which would deserve new perf command.

I can see the analogy with ftrace debugfs interface
- choose/enable tracepoints
- cat .../tracing/trace-pipe

and there could be '-d' for the command to act as daemon.

Right, this is to be integrated into 'trace', i.e. to be able to ask for more events, some with callchains, some without, etc.

We may even have something really bare bones that does what rasd.c does right now, to show how one can write a tool using the exported interfaces in tools/lib/api/, but I think that what rasd is _right now_, is just an attempt to use the libraries using as few as possible apis, right? I.e. more "meat" will be added there, no?

Anyway, its beeing an interesting exercise and will result in a lot of untanglement, which is good in any case.

right.. is there more planned for RAS daemon to do? because at this point it looks to me more like new perf functionality (in trace or some other new command)

Quoted 3 lines

Anyway, its beeing an interesting exercise and will result in a lot of untanglement, which is good in any case.

yea, I dont mind untangling the interface.. I noticed in the thread people want this anyway - simple 'events open' interface.. I often need something simple for test programs

Quoted 8 lines · Jiri Olsa

right.. is there more planned for RAS daemon to do? because at this point it looks to me more like new perf functionality (in trace or some other new command)

Quoted 3 lines

Anyway, its beeing an interesting exercise and will result in a lot of untanglement, which is good in any case.

Quoted 3 lines

yea, I dont mind untangling the interface.. I noticed in the thread people want this anyway - simple 'events open' interface.. I often need something simple for test programs

Right, we simply need a way to get the tracepoint data into userspace. Nothing else. And right now I cannot think about needing any other perf tool functionality. But who knows.., if we need more, we'll untangle more.

:-)

--
Regards/Gruss,
Boris.
--

I read the rasd sources and realized we could poke this from another angle.. AFAIU the work the rasd does is following:
- reads config file and opens configured tracepoints
- reads samples comming from those tracepoints and displays/writes this data to the console/file
- is there more?

If I'm not missing anything, this is quite usefull/common usage pattern which would deserve new perf command.

I can see the analogy with ftrace debugfs interface
- choose/enable tracepoints
- cat .../tracing/trace-pipe

and there could be '-d' for the command to act as daemon.

Quoted 7 lines

I read the rasd sources and realized we could poke this from another angle.. AFAIU the work the rasd does is following:
- reads config file and opens configured tracepoints
- reads samples comming from those tracepoints and displays/writes this data to the console/file
- is there more?

Quoted 8 lines

If I'm not missing anything, this is quite usefull/common usage pattern which would deserve new perf command.

I can see the analogy with ftrace debugfs interface
- choose/enable tracepoints
- cat .../tracing/trace-pipe

and there could be '-d' for the command to act as daemon.

Yes, this parallels a new use case on my end. Right now I am running perf record ... | perf script. With the tracepoints and filters involved it is a LOT of typing - and still collects more than is needed (I don't need MMAP events for example, only COMM events). I am leaning towards a new perf sub-command but from my scheduling timehist and daemon commands I know there is a lot of overhead that goes with that. A perf library with a stable API would make this a lot easier. (AFAIK the python bindings do not currently support opening events, it is mainly an analysis option.)