perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable perf event counters. Here is a demonstration program:

#include <linux/prctl.h>

void loop_events_disabled() {
    volatile int x;
    int i;
    for (i = 0; i < 1000000000; i++)
        x = i;
    return;
}

void loop_events_enabled() {
    volatile int x;
    int i;
    for (i = 0; i < 1000000000; i++)
        x = i;
    return;
}

int main(int argc, char **argv) {
    prctl(PR_TASK_PERF_EVENTS_ENABLE);
    loop_events_enabled();
    prctl(PR_TASK_PERF_EVENTS_DISABLE);
    loop_events_disabled();
}

I would not expect to see loop_events_disabled() show up in the profile as reported by perf report, but it does.

$ perf record ./a.out
$ perf report -n --stdio
# Events: 3K cycles
#
# Overhead  Samples    Command      Shared Object                Symbol
# ........ ..........  .......  .................  ....................
#
    51.80%       1679    a.out  a.out              [.] loop_events_enabled
    48.07%       1578    a.out  a.out              [.] loop_events_disabled
     0.03%          5    a.out  [kernel.kallsyms]  [k] intel_pmu_enable_all
     0.03%          1    a.out  [kernel.kallsyms]  [k] sched_clock
     0.03%          1    a.out  [kernel.kallsyms]  [k] ktime_get
     0.03%          1    a.out  [kernel.kallsyms]  [k] update_cpu_load
     0.01%          1    a.out  [sunrpc]           [k] generic_match

I have tested this on several kernels including 3.3rc1. Can anyone tell me if I'm using this wrong or if this is a bug?

-Andrew

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message.  Thank you.

You're using it wrong, it will disable events you own (created) not events that monitor you.

Is there an alternate way of disabling events that monitor the current process? I ask because I came across the following description in tools/perf/design.txt:

A process can enable or disable all the counter groups that are attached to it, using prctl:

prctl(PR_TASK_PERF_EVENTS_ENABLE);

prctl(PR_TASK_PERF_EVENTS_DISABLE);

This applies to all counters on the current process, whether created by this process or by another, and doesn't affect any counters that this process has created on other processes. It only enables or disables the group leaders, not any other members in the groups.

-Andrew

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message.  Thank you.

Nope, nor will there ever be.

That's wrong.. also I wouldn't ever allow such a 'feature', that's just asking for trouble.

btw., what's your usecase?

Thanks,

Ingo

I'm trying to profile a small section of a long-running program. I ran into trouble using call-graph recording and I thought this might be an alternative way of getting what I was after.

-Andrew

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message.  Thank you.

That usecase indeed makes sense. Peter, could we allow this for privileged tasks, depending on the perf_paranoia settings or such?

Thanks,

Ingo

This sounds useful not only for privileged tasks. Why not make it event attribute? If user wants PR_TASK_PERF_EVENTS_DISABLE be respected for an event it can specify special argument like:

perf record -e cycles:n ./a.out

--
Gleb.

I really dislike it. The sane way around this would be to allow easy self-profiling instead of doing things arse about face like that.

So, what workflow are you suggesting to Andrew?

I guess we are also hurting from the lack of dwarf stack backtrace decoding - that would allow the filtering by parent function without modifying the code. I think Frederic had a prototype working for 32-bit - any update on that?

Andrew could work that problem around right now by adding:

-fno-omit-frame-pointer

to the build of the utility - that should activate -g and perf-report's --parent filter should also work fine.

Thanks,

Ingo

Librarize perf record, then in your code do something like:

#include "perf_record.h"

handle = perf_record_init(); /* creates perf events and creates
a record thread that writes samples to perf.data, consumes env(PERF_*) for configuration, registers with at_exit() for cleanup */
if (!handle)

    /* burn */

/* do you other code */

  perf_record_start(handle);

/* do the bit you want profiled */

  perf_record_stop(handle);

Then build with -lperfrecord or so. Not too hard, right?

Maybe. (and then it shouldnt be limited to perf_record.h but should be events.h plus libevents.so or such)

Isnt a simple prctl() so much easier and faster?

What's your concern with the prctl()? This would arguably be the right kind of usage for prctl(): it's an established API/ABI for process/task-wide settings.

Thanks,

Ingo

Yes it should be, you want to reserve the more generic name for less narrow interfaces.

I really don't want to add another two prctl()s for this, ideally I'd remove the ones we have now, but I've never done due to maintaining backwards blah..

Its doing things backwards, also the whole concept of allowing people to hide things from a profiler is so rotten I'm not willing to even consider the notion.

What does that mean?

But what we want here is not to hide things from the profiler, what we want to be able to is to *ask* the profiler to hide things for us, and we do that for a good reason.

This distinction (which I agree is important) could be implemented by adding a "allow self-profiling" bit (default disabled) in the perf_attr. That way such code would not be able to 'hide' from a simple:

perf top
perf record -a

session, but would be able to self-profile from such a session:

perf record --allow-self-profile ...

where there could be some easy shortcut for
--allow-self-profile, such as:

perf record -S

that way no code could ever hide from a profiler, only if the profiler is specifically allowing self-profiling. (opt in)

The librarization you suggested might make sense too - but i think people will gravitate towards the easier to use variant, and prctl() is as easy and straightforward as it gets.

We can Cc: it to Linus with an explicit [RFC] and he can shoot it down if the API is ugly - but I don't think it's ugly.

Hm?

Thanks,

Ingo

Em Mon, Jan 30, 2012 at 02:45:07PM +0100, Peter Zijlstra escreveu:

Agreed.

- Arnaldo

Em Mon, Jan 30, 2012 at 12:01:33PM +0100, Peter Zijlstra escreveu:

This looks like the test__PERF_RECORD function in tools/perf/builtin-test.c, copying here just the equivalent parts to the above:

	#include "util/evlist.h"

        struct perf_record_opts opts = {
                .target_pid = -1,
                .target_tid = -1,
                .no_delay   = true,
                .freq       = 10,
                .mmap_pages = 256,
                .sample_id_all_avail = true,
        };

        struct perf_evlist *handle = perf_evlist__new(NULL, NULL);

        /* 
         * We need at least one evsel in the evlist, use the default 
         * one: "cycles". 
         */ 
        err = perf_evlist__add_default(evlist);

	perf_evlist__config_attrs(evlist, &opts);

	err = perf_evlist__open(evlist, opts.group);

        perf_evlist__enable(evlist);

	/* do the bit you want profiled */

        perf_evlist__disable(evlist);

----

perf_evlist__config_attrs will, among other chores, do:

        if (opts->target_pid == -1 && opts->target_tid == -1 &&
            !opts->system_wide) {
                attr->disabled = 1;
                attr->enable_on_exec = 1;
        }

And you can fudge it a bit more to your liking before calling perf_evlist__open().

This 'perf test' use case was more for preparing for monitoring
a workload that a program would start, setting up the events, etc, leaving all disabled and then letting the workload begin which would, by means of enable_on_exec start the counters/events.

	But I guess you can just setup evlist->workload.pid to getpid()

and use perf_evlist__enable()/ perf_evlist_disable() pairs to your liking.

Writing a test case that would do exactly what you want in 'perf
test' would be awesome and would allow us to figure out if the abstractions in place are enough for your use case. Then we could librarize just this subset and go from there.

Take a look as well at test__basic_mmap, it may be useful as
another example of using these evsel/evlist classes.

What is needed to link you can find in tools/perf/util/setup.py,
remove util/python.c and you should be set.

- Arnaldo

I haven't touched that for a while. I would be glad if somebody else could relay me on this work.

I think Jiri Olsa has been working on something about Dwarf cfi unwinding with a different approach. Mine was about dumping chunks of stack and regs and do the unwinding on post processing. I think Jiri is doing the unwinding from the event overflow fast path.

Hi,

This thread from a year and a half ago:

http://comments.gmane.org/gmane.linux.kernel/1244915
(or http://permalink.gmane.org/gmane.linux.kernel.perf.user/790
for the initial post)

talks about profiling part of a long-running program using perf, and notes that prctl(PR_TASK_PERF_EVENTS_DISABLE) does not work for this. (By the way, as noted in the thread, its documentation in tools/perf/design.txt is not very helpful - I was initially misled and tried to use it just like the original poster in the thread above.)

The discussion in the thread was inconclusive, but talks about whether there should be better self-profiling support and/or some kind of user-defined contexts for events that could be used to select when events are counted. For a use case and more on the latter, see:

http://permalink.gmane.org/gmane.linux.kernel.perf.user/984

I'd like to point out something that wasn't mentioned in the above
thread: cgroups actually provide a rudimentary way for co-operating user-space code to select when perf should count events and when it should not. The code to be profiled should simply switch itself to a specific cgroup (say "doperf") when it wants perf to count events, and switch back when it doesn't; and perf should be run with -G doperf to count events only in this cgroup. I give a full example below that uses this approach to profile only part of a program.

I also have a few questions:

- Has something changed after the above discussion, i.e., is there some other support in the perf tools to (maybe co-operatively) profile only part of a program? (I did not find any...)

- Does this cgroup approach feel like the proper thing to do, i.e., should I be doing this kind of thing with perf and cgroups at all?

I found very little documentation on this, and I wasn't sure if there would be too much overhead, or even if the perf cgroup support worked like this at all. But it appears to work. (For instance, I feared that -G would just be a shorthand for specifying an explicit list of pids to monitor - but it isn't, since later changes to the list of processes in a given cgroup are noticed.)

- I had to use -a in "perf stat -e branch-misses -a -G doperf ./perftest_cg" below, and thus run perf as root. Without -a I got the error "both cgroup and no-aggregation modes only available in system-wide mode". I guess this is because the -G option profiles the full cgroup and not only the subprocess executed by perf stat. Maybe the API and/or toolset could be changed to allow using -G as a non-root user, perhaps by adding an option to profile a specific pid/command only when it is part of the given cgroup?

(I also know about the PAPI library http://icl.cs.utk.edu/papi/ which is a non-Linux-specific user-space library for self-profiling using the hardware performance counters, and uses the perf kernel API on Linux. It would perhaps be a better choice for self-profiling hardware events like in the example below, but it of course works completely differently from the rather nice perf toolset. So the cgroup approach provides a way to use the perf tools and still select which parts of the program to profile.)

My perf-with-cgroup example follows. I hope I'm using cgroups correctly (I don't really know much about them) - at least it appears to work...

/*
* An example for profiling part of a program by using perf cgroup support
* (needs libcgroup; I used version 0.38 from Debian package libcgroup-dev)
*
* This example demonstrates branch prediction misses caused by
* looping through a random array and branching based on the value of
* each element; since random data cannot be predicted, about one half
* of the predictions should be incorrect. (For an explanation of the
* phenomenon, see <http://stackoverflow.com/questions/11227809>.)
*
* Compile:
* $ gcc -Wall -g -O6 -march=native -DNO_CGROUPS -DNO_TEST_CODE -o perftest_notest perftest.c
* $ gcc -Wall -g -O6 -march=native -DNO_CGROUPS -o perftest_nocg perftest.c
* $ gcc -Wall -g -O6 -march=native -o perftest_cg perftest.c -lcgroup
*
* Test without using cgroups:
* $ perf stat -e branch-misses ./perftest_nocg
* $ perf stat -e branch-misses ./perftest_notest
* (The difference of the two results should be about LOOPSZ/2 =
* 50000000 branch misses. This non-cgroup approach works here, but
* would not work in more complex code, e.g., if the deinitialization
* code acts substantially differently when the actual code to be
* tested is removed.)
*
* Initialize cgroups for test below (# = run as root; change /mnt to whatever):
* # mount -t cgroup -o perf_event myperf /mnt
* # cgcreate -g perf_event:doperf # or just mkdir /mnt/doperf
*
* Test with cgroups (should give about LOOPSZ/2 branch misses):
* # perf stat -e branch-misses -a -G doperf ./perftest_cg
* or (if you want to run perftest_cg as a non-root user):
* # perf stat -e branch-misses -a -G doperf sleep 10
* $ ./perftest_cg
*
* Deinitialize the cgroups:
* # rmdir /mnt/doperf # I can't find a tool like cgcreate for removing
* # umount /mnt
*/

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#ifndef NO_CGROUPS
#include <libcgroup.h>
#endif

#ifndef ASZ
#define ASZ 100000000
#endif

#ifndef LOOPSZ
#define LOOPSZ ASZ
#endif

#ifndef NO_CGROUPS
struct cgroup *cg_doperf, *cg_original;
#endif

int count = 0;

/*
* Use a non-inlined function to force the compiler to create a branch
* that calls it instead of, e.g., a conditional move instruction. */
__attribute__((noinline)) void inc_count(void) {
count++;
}

void do_test(void) {
int i;
int *a = malloc(ASZ * sizeof(int));

  srandom(1);

  for (i = 0; i < ASZ; i++)
    a[i] = (random() > RAND_MAX/2);

/*
* Now everything has been initialized; we are interested in
* profiling only the code that follows.
*/

#ifndef NO_TEST_CODE

#ifndef NO_CGROUPS
cgroup_attach_task(cg_doperf);
#endif

  for (i = 0; i < LOOPSZ; i++)
    if (a[i])
      inc_count();

#ifndef NO_CGROUPS
cgroup_attach_task(cg_original);
#endif

#endif

/*
* End of interesting code; the following should not be profiled. */

printf("%d\n", count); /* the value should be about LOOPSZ/2 */
free(a);
}

int main(void) {
#ifndef NO_CGROUPS
char *cpath = NULL;

  cgroup_init();

  cgroup_get_current_controller_path(getpid(), "perf_event", &cpath);
  cg_original = cgroup_new_cgroup(cpath);
  free(cpath);
  cgroup_add_controller(cg_original, "perf_event");

cg_doperf = cgroup_new_cgroup("doperf");
cgroup_add_controller(cg_doperf, "perf_event");
#endif

  do_test();

#ifndef NO_CGROUPS
cgroup_free(&cg_doperf);
cgroup_free(&cg_original);
#endif

return 0;
}