round-robining per-cpu counters
It used to be, and as far as I can see still is, the case that per-cpu counters take priority over per-task counters by virtue of being scheduled in first. That is, if you have N hardware counters and >= N per-cpu counters, then no per-task counters will ever get scheduled onto the PMU.
That being the case, I don't see what the point of having the perf_reserved_percpu variable is. It doesn't do anything except set cpuctx->max_pertask, which isn't actually used anywhere. In any case with the current counter scheduling system there's no need to "reserve" hardware counters for use by per-cpu counters since any new per-cpu counters will just bump existing per-task counters off - if not immediately then the next time that perf_counter_task_tick gets called.
What was the intended meaning of perf_reserved_percpu? I presume it was that there would always be that many hardware counters available for per-cpu counters regardless of how many per-task counters there are. But that doesn't answer the complementary question - how many hardware counters can we rely on being available for per-task counters? At the moment the answer is 0, but I don't think that is a good answer.
Does anyone have any good ideas about what the scheduling policy should be?
Paul.
Quoted 23 lines · Paul Mackerras
It used to be, and as far as I can see still is, the case that per-cpu counters take priority over per-task counters by virtue of being scheduled in first. That is, if you have N hardware counters and >= N per-cpu counters, then no per-task counters will ever get scheduled onto the PMU.
That being the case, I don't see what the point of having the
perf_reserved_percpuvariable is. It doesn't do anything except setcpuctx->max_pertask, which isn't actually used anywhere. In any case with the current counter scheduling system there's no need to "reserve" hardware counters for use by per-cpu counters since any new per-cpu counters will just bump existing per-task counters off - if not immediately then the next time thatperf_counter_task_tickgets called.What was the intended meaning of
perf_reserved_percpu? I presume it was that there would always be that many hardware counters available for per-cpu counters regardless of how many per-task counters there are. But that doesn't answer the complementary question - how many hardware counters can we rely on being available for per-task counters? At the moment the answer is 0, but I don't think that is a good answer.
Quoted 2 lines
Does anyone have any good ideas about what the scheduling policy should be?
The reservation mechanism really suffered from not being used by anything or anyone, and it thus bit-rotted across 300 follow-on commits.
What would be the primary usecase? Allow admin to set aside (and guarantee) space for task counters? Allow admin to 'force' exclusivity of counter ownership?
I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.
If admin wants stronger ownership of counters then the pinned/exclusive attribute can be used.
We really want to keep the counter-scheduler simple, and we also want to make the default to be as permissive as possible.
Ingo
Quoted 3 lines · Ingo Molnar
The reservation mechanism really suffered from not being used by anything or anyone, and it thus bit-rotted across 300 follow-on commits.
Yeah.
Quoted 8 lines
What would be the primary usecase? Allow admin to set aside (and guarantee) space for task counters? Allow admin to 'force' exclusivity of counter ownership?
I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.
Sounds good, but I'm having trouble visualizing how we combine a fairly static set of per-cpu counters with several sets of per-task counters that come and go as their tasks get scheduled in and out. Is there some clever scheduling technique that sorts all that out?
Quoted 2 lines
If admin wants stronger ownership of counters then the pinned/exclusive attribute can be used.
I agree.
Quoted 2 lines
We really want to keep the counter-scheduler simple, and we also want to make the default to be as permissive as possible.
Definitely.
Paul.
Quoted 10 lines · Paul Mackerras
Quoted 4 lines
I think a better general solution would be to have a single round-robin list for all currently active counters (both percpu and task counters) - and fairly round-robin all of them. The scaling information makes it obvious when this is happening.
Sounds good, but I'm having trouble visualizing how we combine a fairly static set of per-cpu counters with several sets of per-task counters that come and go as their tasks get scheduled in and out. Is there some clever scheduling technique that sorts all that out?
Fair scheduling techniques would import the complexity of kernel/sched*.c into kernel/perf_counter.c - not sure we want that (at this stage at least).
So i'd go for the second best option: the simplest one.
One simple option would be to 'interleave' the two lists on sched-in (take a counter from one list, then from the other, then from the first, etc.), combined with the alternate rotation of both lists.
Another, perhaps even simpler option would be to do the alternation on a per list basis: rotate which of the two lists go first on sched-in, from the task tick.
Can you see any showstopper artifacts in this scheme?
Later on, if anyone finds this method lacking it can be improved gradually either by making the accounting/statistics smarter, or by increasing the frequency of rotation events and coupling it not to the scheduler tick but to a counter perhaps.
Ingo