System/uncore PMUs and unit aggregation
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
Thanks for opening up the discussion on this Will.
For the Qualcomm L2 driver, one objection I had to exposing each unit is that there are so many of them - the minimum starting point is a dozen, so trying to start 9 counters on each means a perf command line specifying 100+ events. Future chips are only likely to increase that.
There is a single CPU node so from an end-user perspective it would seems to make sense to also have a single L2 node. perf already has the ability to create events on multiple units using cpumask, aggregate the results, and split them out per unit with perf stat -a -A, so the user can get that granularity. Exposing separate units would make userspace duplicate a lot of that functionality. This may rely on each uncore unit being associated with a CPU, which is the case with L2.
I agree with your comments regarding groups and I can see that a standard way of representing topology could be useful - in this case, which CPUs are within the same L2 cluster. Perhaps a list of cpumasks, one per L2 unit.
On 11/17/2016 1:17 PM, Will Deacon wrote:
[...]
Quoted 4 lines
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
If I understand your your point 3 correctly, I'm not sure about the need to start and stop them all atomically. That seems to be a tighter requirement than we require for CPU PMUs. For them, perf stat -a creates events/groups on each CPU, then starts and stops them sequentially and sums the results. If that model is acceptable for the CPU to collect and aggregate counts, that should be the same bar that uncore PMUs need to reach. In the L2 case, the driver isn't summing the results, it's still perf doing it, so I may be misinterpreting your comment about where the summation is permitted.
Thanks,
Neil
Signature · 4 lines
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Thanks you Mark and Will to initiate this discussion.
Quoted 24 lines · Will Deacon
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Quoted 7 lines
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
The Hisilicon Hip0x chip has units like L3 cache, Miscellaneous nodes, DDR controller etc.
There are such units in multiple CPU die's in the chip.
The L3 cache is further divided as banks which have separate set of interface (control registers, interrupts etc..).
As per the suggestion, each L3 cache banks will be exposed as a individual PMU instance.
So for e.g. in a board using Hip0x chip with 2 sockets and each socket consists of 2 CPU die,
There will be a total of 16 L3 cache PMU's which will be exposed.
My doubt here is
Each L3 cache PMU has total 22 statistics events. So if registered as a separate PMU, will it not
create multiple entries (with same event names) in event listing for multiple L3 cache PMU's.
Is there a way to avoid this? or this is acceptable?
Just a thought, If we can group them as single PMU and add a config parameter in the event listing to
identify the L3 cache bank(sub unit). e.g: event name will appear as "hisi_l3c2/read_allocate,bank=?/".
And user can choose count from bank 0x01 as -e "hisi_l3c2/read_allocate,bank=0x01/".
And for aggregate count, bank=0xff.
Does it over complicate? Please share your comments.
Quoted 12 lines
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
Quoted 4 lines
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
Does this unit topology and configN method can solve the duplicate event listing issue? Please clarify.
Quoted 7 lines
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
Quoted 5 lines
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
The djtag is a component which connects with some other components in the SoC by Debug Bus.
The registers in components like L3 cache, MN etc are accessed only via djtag.
Please share comments about the confusion. We can discuss to clear them.
Thanks,
Anurup
Quoted 6 lines
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
Quoted 12 lines · Will Deacon
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
Quoted 7 lines
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
This is I think similar to the Intel Uncore situation. We expose every single individual PMU independently. The Intel uncore is wide and varied between parts.
Leaving the rest for Kan, who's doing the Intel uncore bits.
~ Peter
Quoted 48 lines
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
Quoted 15 lines · Will Deacon
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
Quoted 8 lines
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Good to know. Thanks for adressing this on a higher level.
Quoted 7 lines
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
Can you elaborate why they should be treated independent in the later case? What is the problem with going through a list and writing the control register per unit?
Quoted 11 lines
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
Do you mean started/stopped atomically across units?
Quoted 12 lines
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
Quoted 5 lines
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
On Cavium ThunderX the current patches add 4 PMU types, which unfortunately are all handled different. The L2C-TAD and OCX-TLK have control registers per unit. The LMC and L2C-CBC don't have control registers, (free-running counters). So rule 1 might be too restrictive.
I've not looked into groups, would these allow to merge counters from different PMUs in the kernel?
--Jan
Quoted 6 lines
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
Quoted 28 lines · Will Deacon
Quoted 8 lines
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX]
http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com
[Hisilicon Hip0x]
http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at hu
Quoted 8 lines
awei.com [Qualcomm L2]
http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at cod eaurora.orgEach of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
Quoted 7 lines
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
Quoted 47 lines
This is I think similar to the Intel Uncore situation. We expose every single individual PMU independently. The Intel uncore is wide and varied between parts.
Leaving the rest for Kan, who's doing the Intel uncore bits.
~ Peter
Quoted 39 lines
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above.
The individual unit should be a separate PMU. They are standalone system, although they share the same PCI ID.
I think you can use bus:dev:fun to distinguish among them. That is what we did for Skylake and KNL.
I noticed that you pick up random CPU for your uncore PMU. In our implementation, we usually pick up the first available CPU.
Quoted 2 lines
Quoted 2 lines
The Qualcomm L2 looks like it's doing the
right thing and
If there is only one device, it should be OK.
But the variable "num_pmus" make me very confuse.
Quoted 2 lines
Quoted 2 lines
we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
Each djtag device has its own PMU. They name them by scl_id. hisi_l3c*.
Quoted 7 lines
Quoted 7 lines
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
Quoted 5 lines · Jan Glauber
Quoted 3 lines · Will Deacon
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
Quoted 8 lines
Quoted 4 lines
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
Can you elaborate why they should be treated independent in the later case? What is the problem with going through a list and writing the control register per unit?
For one thing, event groups spanning those units cannot be scheduled atomically (some events would be counting while others were not), violating group semantics.
Quoted 6 lines
Quoted 4 lines
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
Do you mean started/stopped atomically across units?
Yes. If some units are counting while others are not, values can be skewed, and therefore potentially misleading.
Quoted 6 lines
Quoted 5 lines
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
Quoted 7 lines
On Cavium ThunderX the current patches add 4 PMU types, which unfortunately are all handled different. The L2C-TAD and OCX-TLK have control registers per unit. The LMC and L2C-CBC don't have control registers, (free-running counters). So rule 1 might be too restrictive.
I've not looked into groups, would these allow to merge counters from different PMUs in the kernel?
No; event groups are strictly single PMU, with the sole exception that software events may be placed inside a hardware event group (since there's no start/stop logic required for SW events).
Thanks,
Mark.
Hi Neil, Anurup, Jan,
Quoted 15 lines · Neil
Thanks for opening up the discussion on this Will.
For the Qualcomm L2 driver, one objection I had to exposing each unit is that there are so many of them - the minimum starting point is a dozen, so trying to start 9 counters on each means a perf command line specifying 100+ events. Future chips are only likely to increase that.
There is a single CPU node so from an end-user perspective it would seems to make sense to also have a single L2 node. perf already has the ability to create events on multiple units using cpumask, aggregate the results, and split them out per unit with perf stat -a -A, so the user can get that granularity. Exposing separate units would make userspace duplicate a lot of that functionality. This may rely on each uncore unit being associated with a CPU, which is the case with L2.
Quoted 3 lines
I agree with your comments regarding groups and I can see that a standard way of representing topology could be useful - in this case, which CPUs are within the same L2 cluster. Perhaps a list of cpumasks, one per L2 unit.
Mark and I had a chat about this earlier today and I think we largely agree with you. That is, for composite PMUs with a notion of CPU affinity for their component units, it makes sense to use the event affinity as a means to address these units, rather than e.g. create separate PMU instances.
However, for PMUs that don't have this notion of affinity, the units should either be exposed individually or, in the case that there is something like shared control logic, they should be addressed through the config fields (e.g. the hisilicon cache with the bank=NN option).
I think this fits with your driver, so please post an updated version addressing Mark's unrelated review comments.
Quoted 7 lines · Will Deacon
[...]
Quoted 4 lines
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
Quoted 8 lines
If I understand your your point 3 correctly, I'm not sure about the need to start and stop them all atomically. That seems to be a tighter requirement than we require for CPU PMUs. For them, perf stat -a creates events/groups on each CPU, then starts and stops them sequentially and sums the results. If that model is acceptable for the CPU to collect and aggregate counts, that should be the same bar that uncore PMUs need to reach. In the L2 case, the driver isn't summing the results, it's still perf doing it, so I may be misinterpreting your comment about where the summation is permitted.
My concern with summation is more that I don't want to expose high-level "meta" events from the driver which in fact end up being a bunch of different events in a bunch of different counters that can't be read atomically. Userspace is free to do that, but the driver shouldn't claim that it can support the event, if you see what I mean?
Will
Quoted 49 lines · Anurup M
Quoted 22 lines · Will Deacon
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Quoted 7 lines
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
The Hisilicon Hip0x chip has units like L3 cache, Miscellaneous nodes, DDR controller etc.
There are such units in multiple CPU die's in the chip.The L3 cache is further divided as banks which have separate set of interface (control registers, interrupts etc..).
As per the suggestion, each L3 cache banks will be exposed as a individual PMU instance.
So for e.g. in a board using Hip0x chip with 2 sockets and each socket consists of 2 CPU die,
There will be a total of 16 L3 cache PMU's which will be exposed.My doubt here is
Each L3 cache PMU has total 22 statistics events. So if registered as a separate PMU, will it not
create multiple entries (with same event names) in event listing for multiple L3 cache PMU's.
Is there a way to avoid this? or this is acceptable?
Quoted 8 lines
Just a thought, If we can group them as single PMU and add a config parameter in the event listing to
identify the L3 cache bank(sub unit). e.g: event name will appear as "hisi_l3c2/read_allocate,bank=?/".
And user can choose count from bank0x01as -e "hisi_l3c2/read_allocate,bank=0x01/".
And for aggregate count, bank=0xff.
Does it over complicate? Please share your comments.
Adding a bank field to the config looks fine to me. I'm assuming the banks aren't CPU affine?
Will
Quoted 5 lines · Will Deacon
Mark and I had a chat about this earlier today and I think we largely agree with you. That is, for composite PMUs with a notion of CPU affinity for their component units, it makes sense to use the event affinity as a means to address these units, rather than e.g. create separate PMU instances.
Quoted 8 lines
However, for PMUs that don't have this notion of affinity, the units should either be exposed individually or, in the case that there is something like shared control logic, they should be addressed through the config fields (e.g. the hisilicon cache with the bank=NN option).
I think this fits with your driver, so please post an updated version addressing Mark's unrelated review comments.
Thanks Will. I'll post a new patch which covers Marks's comments.
Neil
Signature · 4 lines
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Hi Will,
Quoted 24 lines · Will Deacon
Hi all,
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Quoted 7 lines
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
How are we planning to handle multi-node scenario?
if there are X separate PMUs on single socket, are we going to list 2X PMUs on dual socket?
Quoted 31 lines
2. If the units are handled by the same PMU, then care must be taken to handle event groups correctly. That is, if the units cannot be started and stopped atomically, cross-unit groups must be rejected by the driver. Furthermore, any cross-unit scheduling constraints must be honoured so that all the units targetted by a group can schedule the group concurrently.
3. Summing the counters across units is only permitted if the units can all be started and stopped atomically. Otherwise, the counters should be exposed individually. It's up to the driver author to decide what makes sense to sum.
4. Unit topology can optionally be described in sysfs (we should pick some standard directory naming here), and then events targetting specific units can have the unit identifier extracted from the topology encoded in some configN fields.
The million dollar question is: how does that fit in with the drivers I mentioned at the top? Is this overly restrictive, or have we missed stuff?
We certainly want to allow flexibility in the way in which the drivers talk to the hardware, but given that these decisions directly affect the user ABI, some consistent ground rules are required.
For Cavium ThunderX, it's not clear whether or not the individual units could be expressed as separate PMUs, or whether they're caught by one of the rules above. The Qualcomm L2 looks like it's doing the right thing and we can't quite work out what the Hisilicon Hip0x topology looks like, since the interaction with djtag is confusing.
Quoted 7 lines
If the driver authors (on To:) could shed some light on this, then that would be much appreciated!
Thanks,
Will
thanks
Ganapat
Quoted 4 lines
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 26 lines · Ganapatrao Kulkarni
Quoted 25 lines · Will Deacon
We currently have support for three arm64 system PMUs in flight:
[Cavium ThunderX] http://lkml.kernel.org/r/cover.1477741719.git.jglauber at cavium.com [Hisilicon Hip0x] http://lkml.kernel.org/r/1478151727-20250-1-git-send-email-anurup.m at huawei.com [Qualcomm L2] http://lkml.kernel.org/r/1477687813-11412-1-git-send-email-nleeder at codeaurora.org
Each of which have to deal with multiple underlying hardware units in one way or another. Mark and I recently expressed a desire to expose these units to userspace as individual PMU instances, since this can allow:
* Fine-grained control of events from userspace, when you want to see individual numbers as opposed to a summed total
* Potentially ease migration to new SoC revisions, where the units are laid out slightly differently
* Easier handling of cases where the units aren't quite identical
however, this received pushback from all of the patch authors, so there's clearly a problem with this approach. I'm hoping we can try to resolve this here.
Speaking to Mark earlier today, we came up with the following rough rules for drivers that present multiple hardware units as a single PMU:
Quoted 8 lines
Quoted 4 lines
1. If the units share some part of the programming interface (e.g. control registers or interrupts), then they must be handled by the same PMU. Otherwise, they should be treated independently as separate PMU instances.
How are we planning to handle multi-node scenario?
if there are X separate PMUs on single socket, are we going to list 2X PMUs on dual socket?
Sure, why not? Retrofitting multi-node support into a PMU driver sounds pretty messy to me, and I don't see the downside of exposing these as separate instances (which is what they are).
Will