Oprofile Regression Caused by commit e5d1367f17ba6a6fed5fd8b74e4d5720923e0c25 on PPC

This commit seems to have caused a regression with oprofile. It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working. I suspect the problem is that the PMU is being initialized without being reserved for perf. I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 8 lines · Eric B Munson

This commit seems to have caused a regression with oprofile. It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working. I suspect the problem is that the PMU is being initialized without being reserved for perf. I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Quoted 1 line

Eric

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware. When this happens oprofile stops counting. It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile.  It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working.  I suspect the problem is that the PMU is being initialized without being reserved for perf.  I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware.  When this happens oprofile stops counting.  It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

Quoted 13 lines · Peter Zijlstra

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile.  It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working.  I suspect the problem is that the PMU is being initialized without being reserved for perf.  I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 9 lines

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware.  When this happens oprofile stops counting.  It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

Looking at 3.1-rc9, I doubt it's coming from perf_cgroup_switch(). The function is checking for perf context with at least of cgroup event before calling perf_pmu_disable().
If there is no active perf context, there is no cgroup event, thus the function is a nop.
Even if you have a competing perf session, it would have to have at least one cgroup
event for this code to touch HW. The problem must come from somewhere else.

Quoted 1 line

Quoted 23 lines · Stephane Eranian

Quoted 13 lines · Peter Zijlstra

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile.  It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working.  I suspect the problem is that the PMU is being initialized without being reserved for perf.  I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 9 lines

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware.  When this happens oprofile stops counting.  It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

Quoted 11 lines

Looking at 3.1-rc9, I doubt it's coming from perf_cgroup_switch(). The function is checking for perf context with at least of cgroup event before calling perf_pmu_disable().
If there is no active perf context, there is no cgroup event, thus the function is a nop.
Even if you have a competing perf session, it would have to have at least one cgroup
event for this code to touch HW. The problem must come from somewhere else.

Quoted 1 line

Does activating any cgroup setup a cgroup event?

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile. It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working. I suspect the problem is that the PMU is being initialized without being reserved for perf. I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware. When this happens oprofile stops counting. It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile. It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working. I suspect the problem is that the PMU is being initialized without being reserved for perf. I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware. When this happens oprofile stops counting. It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

Quoted 13 lines · Peter Zijlstra

Quoted 12 lines · Eric B Munson

Quoted 9 lines · Eric B Munson

This commit seems to have caused a regression with oprofile.  It is fairly easy to trigger, simply run oprofile monitoring an event that will fire (something frequent like CPU cycles) causes oprofile to fail saying that the PMU is in use. If I disable CONFIG_CGROUP_PERF, everything goes back to working.  I suspect the problem is that the PMU is being initialized without being reserved for perf.  I am not yet sure of the right fix yet so if you have any suggestions I would appreciate them.

Eric

Quoted 9 lines

Quoted 6 lines

This isn't the best description of the behavior we see, what happens is at some point in the profiling session the MMCR register is clobbered by perf_cgroup_switch() which calls perf_pmu_enable() without reserving the PMC hardware.  When this happens oprofile stops counting.  It doesn't happen each time so some runs show event counts that are reasonable, but it can also lead to event counts that are smaller than expected, or completely missing.

What kernel are you testing?

The problem seems to be fixed in 3.1-rc9 so I just need to identify the patch(es) that fixed it.

Eric

Could be:
a8d757e perf events: Fix slow and broken cgroup context switch code

Could be:
a8d757e perf events: Fix slow and broken cgroup context switch code

Quoted 3 lines · Stephane Eranian

Could be:
a8d757e perf events: Fix slow and broken cgroup context switch code

Thanks for the pointer, but the fix was in:
facc4307 perf: Optimize event scheduling locking

This might be a candidate for stable given that oprofile is broken without it.

Quoted 10 lines · Eric B Munson

Quoted 3 lines · Stephane Eranian

Could be:
   a8d757e perf events: Fix slow and broken cgroup context switch code

Thanks for the pointer, but the fix was in:
    facc4307 perf: Optimize event scheduling locking

This might be a candidate for stable given that oprofile is broken without it.

I might feel much more confident about recommending that if someone could explain why that patches fixes what exact problem.

Hi,

I suspect it's because of this chunk:

        rcu_read_lock();

        list_for_each_entry_rcu(pmu, &pmus, entry) {

(quoted diff) +2 -3

-
                cpuctx = this_cpu_ptr(pmu->pmu_cpu_context);
 
-               perf_pmu_disable(cpuctx->ctx.pmu);
-
                /*
                 * perf_cgroup_events says at least one
                 * context on this CPU has cgroup events.

@@ -353,6 +366,8 @@ void perf_cgroup_switch(struct task_struct *task, int mode)
                 * events for a context.
                 */
                if (cpuctx->ctx.nr_cgroups > 0) {
+                       perf_ctx_lock(cpuctx, cpuctx->task_ctx);
+                       perf_pmu_disable(cpuctx->ctx.pmu);
 
 

In other words, you don't call perf_pmu_disable() unless you know you have cgroup events.

Without that, I think you will touch the PMU on cgroup switch and that night conflict with another subsystem using the PMU, e.g. OProfile.

Quoted 15 lines · Eric B Munson

Quoted 10 lines · Stephane Eranian

Quoted 4 lines

Could be:
   a8d757e perf events: Fix slow and broken cgroup context switch code

Thanks for the pointer, but the fix was in:
    facc4307 perf: Optimize event scheduling locking

This might be a candidate for stable given that oprofile is broken without it.

I might feel much more confident about recommending that if someone could explain why that patches fixes what exact problem.