Skip to contentSkip to conversation

round-robining per-cpu counters

DISCUSSION Paul Mackerras 2 participants 4 messages May 5, 2009

lore
Share & tools
lore
#0 lore raw

It used to be, and as far as I can see still is, the case that per-cpu counters take priority over per-task counters by virtue of being scheduled in first. That is, if you have N hardware counters and >= N per-cpu counters, then no per-task counters will ever get scheduled onto the PMU.

That being the case, I don't see what the point of having the perf_reserved_percpu variable is. It doesn't do anything except set cpuctx->max_pertask, which isn't actually used anywhere. In any case with the current counter scheduling system there's no need to "reserve" hardware counters for use by per-cpu counters since any new per-cpu counters will just bump existing per-task counters off - if not immediately then the next time that perf_counter_task_tick gets called.

What was the intended meaning of perf_reserved_percpu? I presume it was that there would always be that many hardware counters available for per-cpu counters regardless of how many per-task counters there are. But that doesn't answer the complementary question - how many hardware counters can we rely on being available for per-task counters? At the moment the answer is 0, but I don't think that is a good answer.

Does anyone have any good ideas about what the scheduling policy should be?

Paul.

#1 lore raw
Quoted 23 lines · Paul Mackerras

It used to be, and as far as I can see still is, the case that per-cpu counters take priority over per-task counters by virtue of being scheduled in first. That is, if you have N hardware counters and >= N per-cpu counters, then no per-task counters will ever get scheduled onto the PMU.

That being the case, I don't see what the point of having the perf_reserved_percpu variable is. It doesn't do anything except set cpuctx->max_pertask, which isn't actually used anywhere. In any case with the current counter scheduling system there's no need to "reserve" hardware counters for use by per-cpu counters since any new per-cpu counters will just bump existing per-task counters off - if not immediately then the next time that perf_counter_task_tick gets called.

What was the intended meaning of perf_reserved_percpu? I presume it was that there would always be that many hardware counters available for per-cpu counters regardless of how many per-task counters there are. But that doesn't answer the complementary question - how many hardware counters can we rely on being available for per-task counters? At the moment the answer is 0, but I don't think that is a good answer.

Quoted 2 lines

Does anyone have any good ideas about what the scheduling policy should be?

The reservation mechanism really suffered from not being used by anything or anyone, and it thus bit-rotted across 300 follow-on commits.

What would be the primary usecase? Allow admin to set aside (and guarantee) space for task counters? Allow admin to 'force' exclusivity of counter ownership?

I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.

If admin wants stronger ownership of counters then the pinned/exclusive attribute can be used.

We really want to keep the counter-scheduler simple, and we also want to make the default to be as permissive as possible.

Ingo

#2 lore raw
Quoted 3 lines · Ingo Molnar

The reservation mechanism really suffered from not being used by anything or anyone, and it thus bit-rotted across 300 follow-on commits.

Yeah.

Quoted 8 lines

What would be the primary usecase? Allow admin to set aside (and guarantee) space for task counters? Allow admin to 'force' exclusivity of counter ownership?

I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.

Sounds good, but I'm having trouble visualizing how we combine a fairly static set of per-cpu counters with several sets of per-task counters that come and go as their tasks get scheduled in and out. Is there some clever scheduling technique that sorts all that out?

Quoted 2 lines

If admin wants stronger ownership of counters then the pinned/exclusive attribute can be used.

I agree.

Quoted 2 lines

We really want to keep the counter-scheduler simple, and we also want to make the default to be as permissive as possible.

Definitely.

Paul.

#3 lore raw
Quoted 10 lines · Paul Mackerras
Quoted 4 lines

I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.

Sounds good, but I'm having trouble visualizing how we combine a fairly static set of per-cpu counters with several sets of per-task counters that come and go as their tasks get scheduled in and out. Is there some clever scheduling technique that sorts all that out?

Fair scheduling techniques would import the complexity of kernel/sched*.c into kernel/perf_counter.c - not sure we want that (at this stage at least).

So i'd go for the second best option: the simplest one.

One simple option would be to 'interleave' the two lists on sched-in (take a counter from one list, then from the other, then from the first, etc.), combined with the alternate rotation of both lists.

Another, perhaps even simpler option would be to do the alternation on a per list basis: rotate which of the two lists go first on sched-in, from the task tick.

Can you see any showstopper artifacts in this scheme?

Later on, if anyone finds this method lacking it can be improved gradually either by making the accounting/statistics smarter, or by increasing the frequency of rotation events and coupling it not to the scheduler tick but to a counter perhaps.

Ingo