Skip to contentSkip to conversation

perf: export registerred pmus via sysfs

RFC PATCH Superseded Lin Ming 10 participants 51 messages May 10, 2010 – May 14, 2010

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lin Ming

author
#0 lore raw

RFC PATCH 3/9 perf: export registerred pmus via sysfs

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

---
 kernel/perf_event.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 46 insertions(+), 0 deletions(-)
kernel/perf_event.c +46 -0
index f19d40e..003b632 100644
@@ -5744,6 +5744,52 @@ static int __init perf_event_sysfs_init(void)
 }
 device_initcall(perf_event_sysfs_init);
 
+#define PMU_ID_LEN 5
+static int __init perf_pmu_sysfs_init(void)
+{
+	struct kobject *pmu_kobj;
+	struct pmu *pmu;
+	struct bin_attribute *bin_attr;
+	char *pmu_id;
+	int err;
+	int i = 0;
+
+	pmu_kobj = kobject_create_and_add("pmus", &cpu_sysdev_class.kset.kobj);
+	if (!pmu_kobj) {
+		printk(KERN_WARNING "%s: kset create error\n", __func__);
+		return -ENOMEM;
+	}
+
+	list_for_each_entry(pmu, &pmus, entry) {
+		bin_attr = kzalloc(sizeof(*bin_attr), GFP_KERNEL);
+		pmu_id = kzalloc(PMU_ID_LEN, GFP_KERNEL);
+
+		if (!bin_attr || !pmu_id) {
+			if (bin_attr)
+				kfree(bin_attr);
+			if (pmu_id)
+				kfree(pmu_id);
+			return -ENOMEM;
+		}
+
+		sysfs_attr_init(&bin_attr.attr);
+		sprintf(pmu_id, "%d\n", i++);
+		bin_attr->size = 0;
+		bin_attr->attr.name = pmu_id;
+		bin_attr->attr.mode = 0444;
+
+		err = sysfs_create_bin_file(pmu_kobj, bin_attr);
+		if (err) {
+			kfree(bin_attr);
+			kfree(pmu_id);
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+}
+device_initcall(perf_pmu_sysfs_init);
+
 void perf_event_register_pmu(struct pmu *pmu)
 {
 	pmu->id = pmu_id_curr++;
#1 lore raw
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

That way we can simply extend it to things like:

/sys/devices/system/cpu/cpuN/nodeN/pmu_id
/sys/devices/system/node/nodeN/pmi_id

Instead of having to add something like:

/sys/devices/system/node/pmus/0..N

author
#2 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

Do you mean /sys/devices/system/cpu/cpuN/pmu_id contain all ids?

For example, each cpu has 4 pmus and the file pmu_id shows something like,

#cat /sys/devices/system/cpu/cpu0/pmu_id
0 1 2 3

....

#cat /sys/devices/system/cpu/cpuN/pmu_id
0 1 2 3

Quoted 11 lines

That way we can simply extend it to things like:

/sys/devices/system/cpu/cpuN/nodeN/pmu_id
/sys/devices/system/node/nodeN/pmi_id

Instead of having to add something like:

/sys/devices/system/node/pmus/0..N

#3 lore raw
Quoted 12 lines · Lin Ming
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

Quoted 7 lines

Do you mean /sys/devices/system/cpu/cpuN/pmu_id contain all ids?

For example, each cpu has 4 pmus and the file pmu_id shows something like,

#cat /sys/devices/system/cpu/cpu0/pmu_id
0 1 2 3

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

It simply doesn't make sense to have multiple PMU IDs on a single topology entry, how would you choose?

author
#4 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Lin Ming
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

Do you mean /sys/devices/system/cpu/cpuN/pmu_id contain all ids?

For example, each cpu has 4 pmus and the file pmu_id shows something like,

Quoted 6 lines
Quoted 2 lines

#cat /sys/devices/system/cpu/cpu0/pmu_id
0 1 2 3

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

Quoted 3 lines

It simply doesn't make sense to have multiple PMU IDs on a single topology entry, how would you choose?

How about
/sys/devices/system/cpu/cpuN/pmu_0
/sys/devices/system/cpu/cpuN/pmu_1
/sys/devices/system/cpu/cpuN/pmu_2
/sys/devices/system/cpu/cpuN/pmu_3
....?

#5 lore raw
Quoted 22 lines · Lin Ming
Quoted 21 lines · Peter Zijlstra
Quoted 19 lines · Lin Ming
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

Do you mean /sys/devices/system/cpu/cpuN/pmu_id contain all ids?

For example, each cpu has 4 pmus and the file pmu_id shows something like,

#cat /sys/devices/system/cpu/cpu0/pmu_id
0 1 2 3

Quoted 7 lines
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

The multiple PMU case still suggests 1 per CPU in most (all?) cases. If you're thinking of PMUs in the northbridge case this would sit under its own topology given that most CPUs will have a shared view of it. Do you have some cases with performance counters in per-CPU memory controllers or something similar?

Quoted 7 lines

How about
/sys/devices/system/cpu/cpuN/pmu_0
/sys/devices/system/cpu/cpuN/pmu_1
/sys/devices/system/cpu/cpuN/pmu_2
/sys/devices/system/cpu/cpuN/pmu_3
....?

If you're following driver model naming conventions, then these should all be pmu.0, pmu.1, etc, etc.

author
#6 lore raw
Quoted 27 lines · Paul Mundt
Quoted 26 lines · Lin Ming
Quoted 24 lines · Peter Zijlstra
Quoted 12 lines · Lin Ming
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Export pmus via sysfs /sys/devices/system/cpu/pmus/0...N The file name is the pmu id, ie, /sys/devices/system/cpu/pmus/N represents pmu id N.
So perf tool can use it to initialize perf_event_attr.

Why create a whole new directory, why not:

/sys/devices/system/cpu/cpuN/pmu_id ?

Quoted 7 lines

Do you mean /sys/devices/system/cpu/cpuN/pmu_id contain all ids?

For example, each cpu has 4 pmus and the file pmu_id shows something like,

#cat /sys/devices/system/cpu/cpu0/pmu_id
0 1 2 3

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

Quoted 6 lines
Quoted 3 lines

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

The multiple PMU case still suggests 1 per CPU in most (all?) cases. If you're thinking of PMUs in the northbridge case this would sit under its own topology given that most CPUs will have a shared view of it. Do you

Take Nehalem core and uncore pmu as an example,

core pmu sits under /sys/devices/system/cpu/cpuN/pmu_id with ID 0,

But uncore pmu is shared by cpus within package, where should it sit under?

Quoted 2 lines

have some cases with performance counters in per-CPU memory controllers or something similar?

Not sure about this now. I'll check.

Quoted 10 lines
Quoted 8 lines

How about
/sys/devices/system/cpu/cpuN/pmu_0
/sys/devices/system/cpu/cpuN/pmu_1
/sys/devices/system/cpu/cpuN/pmu_2
/sys/devices/system/cpu/cpuN/pmu_3
....?

If you're following driver model naming conventions, then these should all be pmu.0, pmu.1, etc, etc.

Thanks,
Lin Ming

#7 lore raw
Quoted 6 lines · Lin Ming

Take Nehalem core and uncore pmu as an example,

core pmu sits under /sys/devices/system/cpu/cpuN/pmu_id with ID 0,

But uncore pmu is shared by cpus within package, where should it sit under?

I already gave you the answer to that:

/sys/devices/system/cpu/cpuN/nodeN/

the uncore pmu is the memory controller (aka node) pmu.

author
#8 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Lin Ming

Take Nehalem core and uncore pmu as an example,

core pmu sits under /sys/devices/system/cpu/cpuN/pmu_id with ID 0,

Quoted 8 lines
Quoted 2 lines

But uncore pmu is shared by cpus within package, where should it sit under?

I already gave you the answer to that:

/sys/devices/system/cpu/cpuN/nodeN/

the uncore pmu is the memory controller (aka node) pmu.

Ah, much clear now.

perf_event_register_pmu needs another paramter, like

perf_event_register_pmu(struct pmu *pmu, int pmu_type);

Thanks.

author
#9 lore raw
Quoted 13 lines · Lin Ming
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines · Lin Ming

Take Nehalem core and uncore pmu as an example,

core pmu sits under /sys/devices/system/cpu/cpuN/pmu_id with ID 0,

But uncore pmu is shared by cpus within package, where should it sit under?

I already gave you the answer to that:

/sys/devices/system/cpu/cpuN/nodeN/

Quoted 7 lines
Quoted 1 line

the uncore pmu is the memory controller (aka node) pmu.

Ah, much clear now.

perf_event_register_pmu needs another paramter, like

perf_event_register_pmu(struct pmu *pmu, int pmu_type);

Or just put this "pmu_type" into struct pmu?

Quoted 2 lines

Thanks.

#10 lore raw
In reply to Lin Ming
Quoted 6 lines · Lin Ming
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

So your registration would indeed look like something:

perf_event_register_pmu(struct pmu *pmu, int type),

where type would normally be -1 (dynamic) but would be PERF_TYPE_ for those already laid down in ABI.

This approach will also give us a good overview
in /sys/class/event_source/, which will be a flat listing of all existing event sources.

Does this make sense?

#11 lore raw
Quoted 60 lines · Peter Zijlstra
Quoted 6 lines · Lin Ming
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...
Quoted 2 lines

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

The GPU folks would hang is off of the drm class or maybe next to it in the PCI space.

#12 lore raw
Quoted 61 lines · Peter Zijlstra
Quoted 60 lines · Peter Zijlstra
Quoted 6 lines · Lin Ming
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...
Quoted 5 lines
Quoted 2 lines

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

The GPU folks would hang is off of the drm class or maybe next to it in the PCI space.

It could conceivably be in multiple places as well - a given event makes sense to enumerate in multiple places.

( For example an 'interrupt' might show up in a given GPU - but it can also
show up amongst the IRQ tracepoints - or something like that. )

But by far the most common case would be for an event source to be attached to one particular place in the sysfs topology.

Note how naturally this scheme extends to all things hardware topology - which is already enumerated in sysfs. It also extends to all things software events in a pretty natural way via /sys/kernel/mm/.

Plus we want to move out the /sys/kernel/debug/ hacks for kprobes and tracepoints into this space as well. (possibly do it with hw-breakpoints as well by attaching them to the CPU directory - for completeness)

That way /sys/class/event_source/ would provide an enumeration of all events to 'perf list' and would automatically be usable by all the perf tooling.

Thanks,

Ingo

#13 lore raw
In reply to Peter Zijlstra

This really needs to loop the ARM PMU guys on this topic. ARM PMU is not something I know very much about, and I really don't understand why I'm on the CC list and not the ARM PMU folk.

I may be mis-stating a problem, but I believe that we have issues with exporting event IDs in that they're SoC implementation specific - but as I say, the ARM PMU guys know the details not me.

I strongly suggest no further discussion without at least looping Will Deacon in on this.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#14 lore raw
In reply to Peter Zijlstra

(Added Will Deacon)

Quoted 65 lines · Peter Zijlstra
Quoted 6 lines · Lin Ming
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

So your registration would indeed look like something:

Quoted 8 lines

perf_event_register_pmu(struct pmu *pmu, int type),

where type would normally be -1 (dynamic) but would be PERF_TYPE_ for those already laid down in ABI.

This approach will also give us a good overview
in /sys/class/event_source/, which will be a flat listing of all existing event sources.

So Russell reminded me that the ARM people have the problem that their /proc/cpuinfo isn't specific enough to map to a unique event map.

Whilst extending ARM /proc/cpuinfo seems like a sensible option it will not cover anything but cpu events.

So in that trend (and to avoid exhaustive in kernel event lists for no good reason), it might make sense to also add some event_source attributes that identify the thing, maybe a event_source_name or event_source_driver field that would allow unique maps to exhaustive event lists.

#15 lore raw

Hi Peter,

Thanks for CC'ing me on this.

Quoted 2 lines · Peter Zijlstra

(Added Will Deacon)

<snip>

Quoted 31 lines
Quoted 31 lines

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

Quoted 8 lines
Quoted 8 lines

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...

I like this approach, but it only makes sense for ARM if we describe the common subset of events [that is, those associated with a PERF_COUNT_] because, although the ARMv7 architecture does define some common event numberings between cores, implementors are at liberty to extend the event space as they wish. These extensions are much better off being handled in userspace.

Another interesting ARM-ism is the vast potential for uncore event sources in SoC devices. The `node' terminology seems a bit confusing here, as there may be counters situated at various points of a bus hierarchy which monitor various types of transactions for example. I suppose these could live under /sys/devices/system/bus/... ? I would expect these kind of counters to be controlled via raw events because having a list of discrete events doesn't really make sense [for example, if I want to count all bursts of a given size, I can encode the burst size into the event number].

Quoted 11 lines

So Russell reminded me that the ARM people have the problem that their /proc/cpuinfo isn't specific enough to map to a unique event map.

Whilst extending ARM /proc/cpuinfo seems like a sensible option it will not cover anything but cpu events.

So in that trend (and to avoid exhaustive in kernel event lists for no good reason), it might make sense to also add some event_source attributes that identify the thing, maybe a event_source_name or event_source_driver field that would allow unique maps to exhaustive event lists.

The ARM perf events backend already has an ID field in the pmu struct which can be used to lookup a string describing what the event source is. Exporting this via sysfs will make it much easier for userspace [and is basically how OProfile deals with PMU identification].

Cheers,

Will

#16 lore raw
Quoted 5 lines · Will Deacon

Another interesting ARM-ism is the vast potential for uncore event sources in SoC devices. The `node' terminology seems a bit confusing here, as there may be counters situated at various points of a bus hierarchy which monitor various types of transactions for example. I suppose these could live under /sys/devices/system/bus/... ?

Right there are more such systems, and yes hooking them into the appropriate machine topology like PCI busses is exactly the intent.

I already mentioned the GPU PMUs living in the appropriate PCI device. But yes, there are far more exotic configurations out there.

Quoted 5 lines

I would
expect these kind of counters to be controlled via raw events because having a list of discrete events doesn't really make sense [for example, if I want to count all bursts of a given size, I can encode the burst size into the event number].

Raw is fine, its only once these things start to converge and show common traits that adding a 'generic' event class becomes useful, and this is something that can always be done later.

#17 lore raw
In reply to Peter Zijlstra
Quoted 35 lines · Peter Zijlstra

[...]

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...
Quoted 2 lines

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

Yeah, we really want a mechanism like this in place instead of continuing with the somewhat ad-hoc extensions to the event enumeration space.

One detail: i think we want one more level. Instead of:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses
                                           /local_hits
                                           /remote_misses
                                           /remote_hits
                                           /...

We want the individual events to be a directory, containing the event_id:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses/event_id
                                           /local_hits/event_id
                                           /remote_misses/event_id
                                           /remote_hits/event_id
                                           /...

The reason is that we want to keep our options open to add more attributes to individual events. (In fact extended attributes already exist for certain event classes - such as the 'format' info for tracepoints.)

Thanks,

Ingo

#18 lore raw
Quoted 24 lines · Ingo Molnar

Yeah, we really want a mechanism like this in place instead of continuing with the somewhat ad-hoc extensions to the event enumeration space.

One detail: i think we want one more level. Instead of:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses
                                           /local_hits
                                           /remote_misses
                                           /remote_hits
                                           /...

We want the individual events to be a directory, containing the event_id:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses/event_id
                                           /local_hits/event_id
                                           /remote_misses/event_id
                                           /remote_hits/event_id
                                           /...
Quoted 3 lines

The reason is that we want to keep our options open to add more attributes to individual events. (In fact extended attributes already exist for certain event classes - such as the 'format' info for tracepoints.)

Sure, sounds like a sensible suggestion.

One thing I'd also like to clarify is that !raw events should not be exhaustive hardware event lists, those are best left for userspace, but instead are generally useful events that can be expected to be implemented by any hardware of that particular class.

So a GPU might have things like 'vsync' and 'cmd_pipeline_stall' or whatever is a generic GPU feature, but not very implementation specific things that the next generation of hardware won't ever have.

#19 lore raw
Quoted 31 lines · Peter Zijlstra
Quoted 24 lines · Ingo Molnar

Yeah, we really want a mechanism like this in place instead of continuing with the somewhat ad-hoc extensions to the event enumeration space.

One detail: i think we want one more level. Instead of:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses
                                           /local_hits
                                           /remote_misses
                                           /remote_hits
                                           /...

We want the individual events to be a directory, containing the event_id:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses/event_id
                                           /local_hits/event_id
                                           /remote_misses/event_id
                                           /remote_hits/event_id
                                           /...
Quoted 3 lines

The reason is that we want to keep our options open to add more attributes to individual events. (In fact extended attributes already exist for certain event classes - such as the 'format' info for tracepoints.)

Sure, sounds like a sensible suggestion.

Quoted 8 lines

One thing I'd also like to clarify is that !raw events should not be exhaustive hardware event lists, those are best left for userspace, but instead are generally useful events that can be expected to be implemented by any hardware of that particular class.

So a GPU might have things like 'vsync' and 'cmd_pipeline_stall' or whatever is a generic GPU feature, but not very implementation specific things that the next generation of hardware won't ever have.

Definitely so.

Ingo

#20 lore raw
Quoted 28 lines · Ingo Molnar
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Ingo Molnar

Yeah, we really want a mechanism like this in place instead of continuing with the somewhat ad-hoc extensions to the event enumeration space.

One detail: i think we want one more level. Instead of:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses
                                           /local_hits
                                           /remote_misses
                                           /remote_hits
                                           /...

We want the individual events to be a directory, containing the event_id:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses/event_id
                                           /local_hits/event_id
                                           /remote_misses/event_id
                                           /remote_hits/event_id
                                           /...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The reason is that we want to keep our options open to add more attributes to individual events. (In fact extended attributes already exist for certain event classes - such as the 'format' info for tracepoints.)

Having extra fields for each event would allow us to describe hardware-specific event attributes. For example:
/sys/devices/system/node/nodeN/node_events
node_events/event_source_id
node_events/local_misses/event_id
/local_hits/event_id
/crypto_datamover <- specific node PMU
/marked_crb_rcv_des
/event_id
/attrib
/lpid <- attribute name
/lpid/type <- type of attribute (boolean, integer, etc.) /lpid/min <- min value of int attribute
/lpid/max <- max value of int attribute
/lpid/bit_offset <- amount to shift attribute value before OR'ing into the raw event code /marking_mode <- attribute name
/marking_mode/type
/...

Of course, these nodes would need to be replicated for each event that needs them or other attributes.

Quoted 7 lines
Quoted 7 lines

Sure, sounds like a sensible suggestion.

One thing I'd also like to clarify is that !raw events should not be exhaustive hardware event lists, those are best left for userspace, but instead are generally useful events that can be expected to be implemented by any hardware of that particular class.

Why exactly is this? I got the impression this was something you and Ingo wanted earlier. As big of an impact as it will be, it would be nice to unify the two event spaces (generic and raw) into one space that can be explored by a user space tool (or even crudely by /bin/ls).

Quoted 8 lines
Quoted 4 lines

So a GPU might have things like 'vsync' and 'cmd_pipeline_stall' or whatever is a generic GPU feature, but not very implementation specific things that the next generation of hardware won't ever have.

Definitely so.

Ingo

Hi Ingo,

In the past, you said that you didn't want to have user space anything enumerate raw hardware events that are supported by the kernel. So does the above represent a re-thinking of that position?

We'd like to have the capability of hardware-specific symbolic event names in the perf tool by some mechanism, unified or otherwise. Right now, for the IBM Wire-Speed processor, we are currently not able to use the perf tool because of its lack of symbolic raw event name support.

In the mean time, we are using a pair of demo programs from Stephane Eranian's libpfm4 source tree called "task" and "syst". These tools use the symbolic event names provided by libpfm4, and use the kernel support from perf_events.

Regards,

- Corey

#21 lore raw
Quoted 18 lines · Corey Ashford

Having extra fields for each event would allow us to describe hardware-specific event attributes. For example:
/sys/devices/system/node/nodeN/node_events
node_events/event_source_id
node_events/local_misses/event_id
/local_hits/event_id
/crypto_datamover <- specific node PMU
/marked_crb_rcv_des
/event_id
/attrib
/lpid <- attribute name
/lpid/type <- type of attribute (boolean, integer, etc.) /lpid/min <- min value of int attribute
/lpid/max <- max value of int attribute
/lpid/bit_offset <- amount to shift attribute value before OR'ing into the raw event code /marking_mode <- attribute name
/marking_mode/type
/...

Quoted 1 line

Of course, these nodes would need to be replicated for each event that needs them or other attributes.

I would suggest having a 1:n pmu:event_source ratio, not the other way around. So in your example above the crypto_datamover PMU thingy would get its own event_source(s), which get placed at whatever place in the machine topology they live. If that happens to be at the node level, place them there.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

Sure, sounds like a sensible suggestion.

One thing I'd also like to clarify is that !raw events should not be exhaustive hardware event lists, those are best left for userspace, but instead are generally useful events that can be expected to be implemented by any hardware of that particular class.

Quoted 5 lines

Why exactly is this? I got the impression this was something you and Ingo wanted earlier. As big of an impact as it will be, it would be nice to unify the two event spaces (generic and raw) into one space that can be explored by a user space tool (or even crudely by /bin/ls).

No we want generic event spaces that have events that are generally applicable to a whole class of PMUs, so say to all CPU PMUs, or all GPU PMUs or all PCI-bridge PMUs.

What we do not want are exhaustive event lists for specific PMU implementations, those are best left for userspace.

I'll let Ingo answer your other question.

#22 lore raw
Quoted 2 lines · Peter Zijlstra

What we do not want are exhaustive event lists for specific PMU implementations, those are best left for userspace.

I'd refine this the following way:

- We mandate proper in-kernel enumeration of all things event sources, for example /sys/devices/system/node/nodeN/node_events. Obviously an event source needs to be addressable for it to be useful to userspace.

- We want generalized events expressed in those event containers that are used commonly. Whatever people find useful we can enumerate and what is enumerated is an ABI.

- The 'rest' can go into /sys/devices/system/node/nodeN/node_events/raw_event/. These will never be guaranteed in an ABI way really (although will work in some cases) - those using raw event codes are really up to themselves and if it ever gets in the way of proper, more expressive enumeration/generalization it will have to yield.

These are the ground rules as i see them.

Ingo

author
#23 lore raw
Quoted 21 lines · Ingo Molnar
Quoted 2 lines · Peter Zijlstra

What we do not want are exhaustive event lists for specific PMU implementations, those are best left for userspace.

I'd refine this the following way:

- We mandate proper in-kernel enumeration of all things event sources, for example /sys/devices/system/node/nodeN/node_events. Obviously an event source needs to be addressable for it to be useful to userspace.

- We want generalized events expressed in those event containers that are used commonly. Whatever people find useful we can enumerate and what is enumerated is an ABI.

- The 'rest' can go into /sys/devices/system/node/nodeN/node_events/raw_event/. These will never be guaranteed in an ABI way really (although will work in some cases) - those using raw event codes are really up to themselves and if it ever gets in the way of proper, more expressive enumeration/generalization it will have to yield.

Quoted 3 lines

These are the ground rules as i see them.

Ingo

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

5. other setting...

6. call syscall perf_event_open(....)

#24 lore raw
Quoted 18 lines · Lin Ming

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

Quoted 8 lines

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

5. other setting...

6. call syscall perf_event_open(....)

No, you'll use event_source_id as perf_event_attr::type, use event_id as perf_event_attr::config and then use a cpu-wide counter on a cpu contained in node0.

author
#25 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Lin Ming

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

Quoted 7 lines
Quoted 3 lines

5. other setting...

6. call syscall perf_event_open(....)

No, you'll use event_source_id as perf_event_attr::type, use event_id as perf_event_attr::config and then use a cpu-wide counter on a cpu contained in node0.

Is event_source_id a link to event_source class?

For example, 2 event sources on Nehalem
/sys/class/event_sources/core_pmu
/sys/class/event_sources/uncore_pmu

Then, /sys/devices/system/node/nodeN/node_events/event_source_id is a link to /sys/class/event_sources/uncore_pmu.

/sys/devices/system/cpu/cpuN/cpu_hardware_events/event_source_id is a link to /sys/class/event_sources/core_pmu.

And, the event_source_id
#cat /sys/class/event_sources/core_pmu
0

#cat /sys/class/event_sources/uncore_pmu
1

author
#26 lore raw
Quoted 47 lines · Lin Ming
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Lin Ming

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

Quoted 7 lines
Quoted 3 lines

5. other setting...

6. call syscall perf_event_open(....)

No, you'll use event_source_id as perf_event_attr::type, use event_id as perf_event_attr::config and then use a cpu-wide counter on a cpu contained in node0.

Is event_source_id a link to event_source class?

For example, 2 event sources on Nehalem
/sys/class/event_sources/core_pmu
/sys/class/event_sources/uncore_pmu

Then, /sys/devices/system/node/nodeN/node_events/event_source_id is a link to /sys/class/event_sources/uncore_pmu.

/sys/devices/system/cpu/cpuN/cpu_hardware_events/event_source_id is a link to /sys/class/event_sources/core_pmu.

Quoted 6 lines

And, the event_source_id
#cat /sys/class/event_sources/core_pmu
0

#cat /sys/class/event_sources/uncore_pmu
1

Oh, no.

/sys/class/event_sources/uncore_pmu is not 1, but other dynamic value.

#27 lore raw
In reply to Lin Ming
Quoted 28 lines · Lin Ming
Quoted 27 lines · Peter Zijlstra
Quoted 25 lines · Lin Ming

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

5. other setting...

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

6. call syscall perf_event_open(....)

No, you'll use event_source_id as perf_event_attr::type, use event_id as perf_event_attr::config and then use a cpu-wide counter on a cpu contained in node0.

Is event_source_id a link to event_source class?

No its an attribute of said class.

Quoted 10 lines

For example, 2 event sources on Nehalem
/sys/class/event_sources/core_pmu
/sys/class/event_sources/uncore_pmu

Then, /sys/devices/system/node/nodeN/node_events/event_source_id is a link to /sys/class/event_sources/uncore_pmu.

/sys/devices/system/cpu/cpuN/cpu_hardware_events/event_source_id is a link to /sys/class/event_sources/core_pmu.

The other way around, look in /sys/class/*/, they're all symlinks.

Quoted 6 lines

And, the event_source_id
#cat /sys/class/event_sources/core_pmu
0

#cat /sys/class/event_sources/uncore_pmu
1

You can't cat a directory. You'd cat something
like: /sys/class/event_sources/core_pmu/event_source_id

And they would contain PERF_TYPE_* like things.

So for the current CPU PMUs we'd already create 3 event classes, cpu_hw_events, cpu_hw_cache_events, cpu_raw_events, with resp. event_source_id of 0, 3 and 4.

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

#28 lore raw
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

Thanks,

Ingo

#29 lore raw
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

Quoted 8 lines

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

Whatever, that's almost identical. What we can do is reserve perf_event_attr::type with bit 31 set for fd's and use the fd->file lookup instead of the type->pmu lookup.

#30 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

Quoted 7 lines
Quoted 5 lines

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

Whatever, that's almost identical. [...]

It's not identical: as we dont expose our mapping structure externally and dont have to have some dynamic type ID allocation layer/mechanism. Also, using fds is an elegant, Linuxish way of expressing some object's identity and passing it along.

( It also removes the possibility to intentionally or accidentally have type
IDs that are not reachable via the sysfs and vica verse. )

Quoted 2 lines

[...] What we can do is reserve perf_event_attr::type with bit 31 set for fd's and use the fd->file lookup instead of the type->pmu lookup.

Yeah, that sounds good.

Ingo

author
#31 lore raw
Quoted 16 lines · Ingo Molnar
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

Quoted 6 lines
Quoted 1 line

Whatever, that's almost identical. [...]

It's not identical: as we dont expose our mapping structure externally and dont have to have some dynamic type ID allocation layer/mechanism. Also, using fds is an elegant, Linuxish way of expressing some object's identity and passing it along.

Don't understand this well enough.

Does it need to attach pmu info to the sys file?

For example,

Does "perf top -e cycles" work as below?

1. get the pmu file of cycles events,
ie, /sys/devices/system/cpu/cpu_hw_events/event_source_id

2. fd = open("/sys/devices/system/cpu/cpu_hw_events/event_source_id")

3. perf_event_attr::type = fd | (1<<31)

4. In kernel, perf_event_lookup_pmu() finds pmu by fd -> sys file -> pmu

Quoted 10 lines

( It also removes the possibility to intentionally or accidentally have type
IDs that are not reachable via the sysfs and vica verse. )

Quoted 2 lines

[...] What we can do is reserve perf_event_attr::type with bit 31 set for fd's and use the fd->file lookup instead of the type->pmu lookup.

Yeah, that sounds good.

Ingo

author
#32 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

Quoted 8 lines

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

I still don't understand this sys_fd -> pmu lookup, would you please explain it more detail?

struct pmu {
	kobject kobj;
	...
};

What I can imagine is,

1. In userspace, sys_fd = open("/sys/devices/system/cpu/event_source", ..), then sys_fd is passed to sys_perf_event_open()

2. In kernel, sys_file = <find the sys file structure with sys_fd>

3. kobject = <retrieve the kobject from sys_file>

4. pmu = container_of(kobject, struct pmu, kobj)

If my understanding is correct, then step 3 above seems strange. It's not the typical usage of sys file.

Thanks,
Lin Ming

#33 lore raw
Quoted 22 lines · Lin Ming
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

Quoted 8 lines

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

I still don't understand this sys_fd -> pmu lookup, would you please explain it more detail?

struct pmu {
	kobject kobj;
	...
};
Quoted 5 lines

What I can imagine is,

1. In userspace, sys_fd = open("/sys/devices/system/cpu/event_source", ..), then sys_fd is passed to sys_perf_event_open()

Yes, open() an event_source - or rather an event itself. For raw events there has to be a separate event entry that can be opened.

I.e. we'd have a layout like:

/sys/devices/system/cpu/events/cycles/id
/sys/devices/system/cpu/events/instructions/id
/sys/devices/system/cpu/events/raw/id

By making each event category a directory we gain the flexibility of integrating tracepoints as well, for example:

/sys/kernel/sched/events/wakeup/id
/sys/kernel/sched/events/wakeup/format

Where 'format' describes the event record layout:

 # cat /debug/tracing/events/sched/sched_wakeup/format 
 name: sched_wakeup
 ID: 59
 format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;
	field:int common_lock_depth;	offset:8;	size:4;	signed:1;
	field:char comm[TASK_COMM_LEN];	offset:12;	size:16;	signed:1;
	field:pid_t pid;	offset:28;	size:4;	signed:1;
	field:int prio;	offset:32;	size:4;	signed:1;
	field:int success;	offset:36;	size:4;	signed:1;
	field:int target_cpu;	offset:40;	size:4;	signed:1;

print fmt: "comm=%s pid=%d prio=%d success=%d target_cpu=%03d", REC->comm, REC->pid, REC->prio, REC->success, REC->target_cpu

Quoted 8 lines

2. In kernel, sys_file = <find the sys file structure with sys_fd>

3. kobject = <retrieve the kobject from sys_file>

4. pmu = container_of(kobject, struct pmu, kobj)

If my understanding is correct, then step 3 above seems strange. It's not the typical usage of sys file.

I dont think it's stange - we demux from the generic sysfs object to the more specific perf events related object. This is similar how driver specific sysfs functionality does the demux as well.

Ingo

author
#34 lore raw
Quoted 69 lines · Ingo Molnar
Quoted 22 lines · Lin Ming
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

Quoted 8 lines

I dont think we should use a dynamic range of event sources - it's a completely useless indirection that has no meaning to humans.

As far as machine interfaces go a much cleaner approach would be to allow an open fd to a sysfs file to be passed to sys_perf_event_open() - this would identify the event source. This needs a small extension of the ABI but we could thus get rid of the 'type' enumeration altogether and express _all_ event sources via fds to sysfs files.

I still don't understand this sys_fd -> pmu lookup, would you please explain it more detail?

struct pmu {
	kobject kobj;
	...
};
Quoted 5 lines

What I can imagine is,

1. In userspace, sys_fd = open("/sys/devices/system/cpu/event_source", ..), then sys_fd is passed to sys_perf_event_open()

Yes, open() an event_source - or rather an event itself. For raw events there has to be a separate event entry that can be opened.

I.e. we'd have a layout like:

/sys/devices/system/cpu/events/cycles/id
/sys/devices/system/cpu/events/instructions/id
/sys/devices/system/cpu/events/raw/id

By making each event category a directory we gain the flexibility of integrating tracepoints as well, for example:

/sys/kernel/sched/events/wakeup/id
/sys/kernel/sched/events/wakeup/format

Where 'format' describes the event record layout:

 # cat /debug/tracing/events/sched/sched_wakeup/format 
 name: sched_wakeup
 ID: 59
 format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;
	field:int common_lock_depth;	offset:8;	size:4;	signed:1;
	field:char comm[TASK_COMM_LEN];	offset:12;	size:16;	signed:1;
	field:pid_t pid;	offset:28;	size:4;	signed:1;
	field:int prio;	offset:32;	size:4;	signed:1;
	field:int success;	offset:36;	size:4;	signed:1;
	field:int target_cpu;	offset:40;	size:4;	signed:1;

print fmt: "comm=%s pid=%d prio=%d success=%d target_cpu=%03d", REC->comm, REC->pid, REC->prio, REC->success, REC->target_cpu

Quoted 4 lines

2. In kernel, sys_file = <find the sys file structure with sys_fd>

3. kobject = <retrieve the kobject from sys_file>

Quoted 8 lines
Quoted 4 lines

4. pmu = container_of(kobject, struct pmu, kobj)

If my understanding is correct, then step 3 above seems strange. It's not the typical usage of sys file.

I dont think it's stange - we demux from the generic sysfs object to the more specific perf events related object. This is similar how driver specific sysfs functionality does the demux as well.

OK, thanks for the explanation.

author
#35 lore raw
In reply to Peter Zijlstra
Quoted 43 lines · Peter Zijlstra
Quoted 28 lines · Lin Ming
Quoted 27 lines · Peter Zijlstra
Quoted 25 lines · Lin Ming

How will this sysfs interface be used for userspace tool?

/sys/devices/system/node/nodeN/node_events
node_events/event_source_id

node_events/local_misses/event_id
/local_hits/event_id /remote_misses/event_id /remote_hits/event_id

For example, to monitor node event local_misses on node 0, does it work as below?

1. perf top -e local_misses -n 0 (-n 0 means node 0)

2. read /sys/devices/system/node/node0/node_events/event_source_id to get the pmu_id

3. read /sys/devices/system/node/node0/node_events/local_misses/event_id to get the event_id

4. event_attr::pmu_id=pmu_id, event::config=event_id

5. other setting...

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

6. call syscall perf_event_open(....)

No, you'll use event_source_id as perf_event_attr::type, use event_id as perf_event_attr::config and then use a cpu-wide counter on a cpu contained in node0.

Is event_source_id a link to event_source class?

No its an attribute of said class.

Quoted 4 lines

For example, 2 event sources on Nehalem
/sys/class/event_sources/core_pmu
/sys/class/event_sources/uncore_pmu

Quoted 8 lines
Quoted 6 lines

Then, /sys/devices/system/node/nodeN/node_events/event_source_id is a link to /sys/class/event_sources/uncore_pmu.

/sys/devices/system/cpu/cpuN/cpu_hardware_events/event_source_id is a link to /sys/class/event_sources/core_pmu.

The other way around, look in /sys/class/*/, they're all symlinks.

/sys/devices/system/cpu/cpu0/cpu_hw_events/* /sys/devices/system/cpu/cpu0/cpu_hw_cache_events/* /sys/devices/system/cpu/cpu0/cpu_raw_events/*
....
....
/sys/devices/system/cpu/cpuN/cpu_hw_events/* /sys/devices/system/cpu/cpuN/cpu_hw_cache_events/* /sys/devices/system/cpu/cpuN/cpu_raw_events/*

Is /sys/class/event_sources/* looks like,

/sys/class/event_sources/cpu_hw_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

/sys/class/event_sources/cpu_hw_cache_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_cache_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

?

Quoted 18 lines
Quoted 7 lines

And, the event_source_id
#cat /sys/class/event_sources/core_pmu
0

#cat /sys/class/event_sources/uncore_pmu
1

You can't cat a directory. You'd cat something
like: /sys/class/event_sources/core_pmu/event_source_id

And they would contain PERF_TYPE_* like things.

So for the current CPU PMUs we'd already create 3 event classes, cpu_hw_events, cpu_hw_cache_events, cpu_raw_events, with resp. event_source_id of 0, 3 and 4.

The new PMUs will use a dynamic range that starts at PERF_TYPE_MAX.

#36 lore raw
Quoted 17 lines · Lin Ming

/sys/devices/system/cpu/cpu0/cpu_hw_events/* /sys/devices/system/cpu/cpu0/cpu_hw_cache_events/* /sys/devices/system/cpu/cpu0/cpu_raw_events/*
....
....
/sys/devices/system/cpu/cpuN/cpu_hw_events/* /sys/devices/system/cpu/cpuN/cpu_hw_cache_events/* /sys/devices/system/cpu/cpuN/cpu_raw_events/*

Is /sys/class/event_sources/* looks like,

/sys/class/event_sources/cpu_hw_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 5 lines

/sys/class/event_sources/cpu_hw_cache_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_cache_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Hmm, good question.

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

Possibly we'd have to place them at the cpu level, like:

/sys/devices/system/cpu/cpu_*_events/

and have links like:

/sys/devices/system/cpu/cpuN/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

as well as

/sys/class/event_sources/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

author
#37 lore raw
Quoted 35 lines · Peter Zijlstra
Quoted 17 lines · Lin Ming

/sys/devices/system/cpu/cpu0/cpu_hw_events/* /sys/devices/system/cpu/cpu0/cpu_hw_cache_events/* /sys/devices/system/cpu/cpu0/cpu_raw_events/*
....
....
/sys/devices/system/cpu/cpuN/cpu_hw_events/* /sys/devices/system/cpu/cpuN/cpu_hw_cache_events/* /sys/devices/system/cpu/cpuN/cpu_raw_events/*

Is /sys/class/event_sources/* looks like,

/sys/class/event_sources/cpu_hw_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 5 lines

/sys/class/event_sources/cpu_hw_cache_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_cache_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Hmm, good question.

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

Possibly we'd have to place them at the cpu level, like:

/sys/devices/system/cpu/cpu_*_events/

and have links like:

Quoted 7 lines

/sys/devices/system/cpu/cpuN/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

as well as

/sys/class/event_sources/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

Ah, I like this idea.

Thanks.

#38 lore raw
In reply to Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 17 lines · Lin Ming

/sys/devices/system/cpu/cpu0/cpu_hw_events/* /sys/devices/system/cpu/cpu0/cpu_hw_cache_events/* /sys/devices/system/cpu/cpu0/cpu_raw_events/*
....
....
/sys/devices/system/cpu/cpuN/cpu_hw_events/* /sys/devices/system/cpu/cpuN/cpu_hw_cache_events/* /sys/devices/system/cpu/cpuN/cpu_raw_events/*

Is /sys/class/event_sources/* looks like,

/sys/class/event_sources/cpu_hw_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 5 lines

/sys/class/event_sources/cpu_hw_cache_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_cache_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 8 lines

Hmm, good question.

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

Possibly we'd have to place them at the cpu level, like:

/sys/devices/system/cpu/cpu_*_events/

The problem with this is /sys/devices/system/ are the horrid sysdev structures, which don't play nice (or at all) with the rest of the driver/device model.

If we fix them up to finally work properly like real devices then we could do this:

Quoted 9 lines

and have links like:

/sys/devices/system/cpu/cpuN/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

as well as

/sys/class/event_sources/cpu_*_events ->
/sys/devices/system/cpu/cpu_*_events/

Like you want to have done.

{sigh}

I guess I'll finally have to start working on this. I had some conversations at the last Collab Summit with someone on how to properly fix this all up, I'll go dig for that note and start in on it.

thanks,

greg k-h

#39 lore raw
In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming

Is /sys/class/event_sources/* looks like,

/sys/class/event_sources/cpu_hw_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 5 lines

/sys/class/event_sources/cpu_hw_cache_events0
-> /sys/devices/system/cpu/cpu0/cpu_hw_events
...
/sys/class/event_sources/cpu_hw_cache_eventsN
-> /sys/devices/system/cpu/cpuN/cpu_hw_events

Quoted 5 lines

Hmm, good question.

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

This is something I've been thinking about, too. On SH we have a large set of perf counter events that are entirely dependent on the configuration of the CPU they're on, with no requirement that these configurations are identical on all CPUs in an SMP configuration.

As an example, it's possible to halve the L1 dcache and use that part of it as a small and fast memory which has completely different events associated with it from the regular L1 dcache events. These events would be invalid on a CPU that was running with all cache ways enabled but might also be valid on other CPUs that bolt these events to an extra SRAM outside of the cache topology completely.

In any event, the events are at least consistent across all CPUs, it's only which ones are valid on a given CPU at a given time that can change.

#40 lore raw
Quoted 17 lines · Paul Mundt
Quoted 3 lines · Peter Zijlstra

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

This is something I've been thinking about, too. On SH we have a large set of perf counter events that are entirely dependent on the configuration of the CPU they're on, with no requirement that these configurations are identical on all CPUs in an SMP configuration.

As an example, it's possible to halve the L1 dcache and use that part of it as a small and fast memory which has completely different events associated with it from the regular L1 dcache events. These events would be invalid on a CPU that was running with all cache ways enabled but might also be valid on other CPUs that bolt these events to an extra SRAM outside of the cache topology completely.

Quoted 2 lines

In any event, the events are at least consistent across all CPUs, it's only which ones are valid on a given CPU at a given time that can change.

So you're running with asymmetric SMP systems? I really hadn't considered that. Will this change at runtime or is it a system boot time thing?

#41 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Paul Mundt
Quoted 3 lines · Peter Zijlstra

No all the cpus would have the same event sources. I'm not sure if we can make sysfs understand that though (added GregKH and Kay to CC).

This is something I've been thinking about, too. On SH we have a large set of perf counter events that are entirely dependent on the configuration of the CPU they're on, with no requirement that these configurations are identical on all CPUs in an SMP configuration.

As an example, it's possible to halve the L1 dcache and use that part of it as a small and fast memory which has completely different events associated with it from the regular L1 dcache events. These events would be invalid on a CPU that was running with all cache ways enabled but might also be valid on other CPUs that bolt these events to an extra SRAM outside of the cache topology completely.

Quoted 6 lines
Quoted 2 lines

In any event, the events are at least consistent across all CPUs, it's only which ones are valid on a given CPU at a given time that can change.

So you're running with asymmetric SMP systems? I really hadn't considered that. Will this change at runtime or is it a system boot time thing?

At the moment it's a boot time thing, but we're moving towards runtime switching via CPU hotplug (which at the moment we primarily use for runtime power management). This has specifically been a recurring requirement from some of our automotive customers, so it's gradually becoming more prevalent.

We also have the multi-core case where multiple architectures are combined but we still have memory-mapped access to the slave CPUs performance counters (SH-Mobile G series has this behaviour where we have both an ARM and an SH core and it doesn't really matter which one is running the primary linux kernel, the slave on the other hand might be running linux or it may be running a fixed application that depends on control and input from the primary linux-running MPU). Presently we just tie in through the hardware debug interfaces for monitoring and controlling the secondary counters, but being able to make this sort of thing workload granular via perf would obviously be a huge benefit. Supporting these sorts of configurations is going to take a bit of doing though, especially on the topology side.

#42 lore raw
In reply to Corey Ashford

Corey,

Quoted 4 lines · Corey Ashford

We'd like to have the capability of hardware-specific symbolic event names in the perf tool by some mechanism, unified or otherwise.  Right now, for the IBM Wire-Speed processor, we are currently not able to use the perf tool because of its lack of symbolic raw event name support.

In the mean time, we are using a pair of demo programs from Stephane Eranian's libpfm4 source tree called "task" and "syst".  These tools use the symbolic event names provided by libpfm4, and use the kernel support from perf_events.

I will soon post a patch to make use of libpfm4 in perf, thereby giving access to all events
unit masks, filters, using symbolic names.

#43 lore raw
In reply to Ingo Molnar

From: Ingo Molnar <[email protected]>
Date: Mon, May 10, 2010 at 01:43:11PM +0200

Hi all,

Quoted 23 lines

Yeah, we really want a mechanism like this in place instead of continuing with the somewhat ad-hoc extensions to the event enumeration space.

One detail: i think we want one more level. Instead of:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses
                                           /local_hits
                                           /remote_misses
                                           /remote_hits
                                           /...

We want the individual events to be a directory, containing the event_id:

/sys/devices/system/node/nodeN/node_events

                                node_events/event_source_id
                                node_events/local_misses/event_id
                                           /local_hits/event_id
                                           /remote_misses/event_id
                                           /remote_hits/event_id
                                           /...
Quoted 3 lines

The reason is that we want to keep our options open to add more attributes to individual events. (In fact extended attributes already exist for certain event classes - such as the 'format' info for tracepoints.)

ok, what you guys have so far sounds ok, here's some more stuff we should be considering when using the tracepoints (and their representation in /sysfs or whatever) for error reporting.

All the error reporting is done using MCEs so the
MCE should be a raw per cpu event somewhere under /sys/devices/system/cpu/cpuN/events/raw_cpu_events/ or whatever works for you.

Another point I have is that MCEs don't need pmus so we should consider having the ability to decouple events from pmus.

What you basically want to have is a tracepoint which is "persistent," as Ingo suggested earlier, and it buffers MCEs occurring at any time into a ring buffer until a userspace daemon or similar sucks that data out for processing (critical stuff is handled differently, of course). And this should work on any x86 hw supporting MCA without hw perf monitoring features.

Also, we might think in terms of using some of the MCE fields in /sysfs for hardware error injection like EDAC does inject DRAM ECC errors but this should be straight-forward using one attribute like

/sys/devices/system/cpu/cpuN/events/raw_cpu_events/mce/inject_ecc

or similar.

This is mostly what I can come up with now...

Signature · 6 lines
Regards/Gruss,
Boris.

--
Advanced Micro Devices, Inc.
Operating Systems Research Center
#44 lore raw
Quoted 2 lines · Borislav Petkov

Another point I have is that MCEs don't need pmus so we should consider having the ability to decouple events from pmus.

Strictly speaking tracepoints are software events, which run off of a software 'pmu'. So no, we can't decouple, they need a 'pmu' context.

Quoted 6 lines

What you basically want to have is a tracepoint which is "persistent," as Ingo suggested earlier, and it buffers MCEs occurring at any time into a ring buffer until a userspace daemon or similar sucks that data out for processing (critical stuff is handled differently, of course). And this should work on any x86 hw supporting MCA without hw perf monitoring features.

Try building x86 without perf hw support :/

#45 lore raw

From: Peter Zijlstra <[email protected]>
Date: Tue, May 11, 2010 at 10:25:15AM -0400

Quoted 6 lines · Borislav Petkov
Quoted 2 lines

Another point I have is that MCEs don't need pmus so we should consider having the ability to decouple events from pmus.

Strictly speaking tracepoints are software events, which run off of a software 'pmu'. So no, we can't decouple, they need a 'pmu' context.

We could make this configurable depending on the severity of the error. I'm guessing for further event handling through the perf infrastructure we cannot run without a sw pmu context but on critical conditions we need to run as fast and as sparingly as possible so I'm thinking maybe adding some specially tailored callbacks to the MCE tracepoint trace_mce_record, as Steven suggested.

Quoted 8 lines
Quoted 6 lines

What you basically want to have is a tracepoint which is "persistent," as Ingo suggested earlier, and it buffers MCEs occurring at any time into a ring buffer until a userspace daemon or similar sucks that data out for processing (critical stuff is handled differently, of course). And this should work on any x86 hw supporting MCA without hw perf monitoring features.

Try building x86 without perf hw support :/

I dunno, maybe decoupling wouldn't be necessary per se but I was simply pointing out that MCEs shouldn't necessarily depend on the presence on hardware performance counters.

Signature · 6 lines
Regards/Gruss,
Boris.

--
Advanced Micro Devices, Inc.
Operating Systems Research Center
#46 lore raw
Quoted 10 lines · Borislav Petkov

From: Peter Zijlstra <[email protected]>
Date: Tue, May 11, 2010 at 10:25:15AM -0400

Quoted 6 lines · Borislav Petkov
Quoted 2 lines

Another point I have is that MCEs don't need pmus so we should consider having the ability to decouple events from pmus.

Strictly speaking tracepoints are software events, which run off of a software 'pmu'. So no, we can't decouple, they need a 'pmu' context.

Quoted 6 lines

We could make this configurable depending on the severity of the error. I'm guessing for further event handling through the perf infrastructure we cannot run without a sw pmu context but on critical conditions we need to run as fast and as sparingly as possible so I'm thinking maybe adding some specially tailored callbacks to the MCE tracepoint trace_mce_record, as Steven suggested.

Well, all the tracepoint stuff should already be NMI-safe (all of perf events needs to be because the PMI is an NMI) and I think perf as a whole would like to run as fast as possible, so I don't yet see the need for special purpose hooks (which I'll try to resist as much as possible).

#47 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Lin Ming
Quoted 2 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though,

:-)

Quoted 48 lines
Quoted 5 lines
Quoted 2 lines

but I think the sanest way is to extend the CPU
topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

Quoted 7 lines

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...

Just to give a concrete example, the IBM Wire-Speed Processor has four AT-"nodes" per chip, each containing four PowerPC cores.

Those four nodes together share a number of nest PMU accelerators, I/O devices, buses etc. which each have their own PMUs. Further adding to the structure is that some of the nodes are replicated. For example, we have two memory controllers, each with a pair of PMUs.

/sys/devices/system/node/node0/mem_ctlr0/

                                        event_source_id
                                        events/
                                              partial_cacheline_read_retried/
                                              partial_cacheline_write_retried/
                                              ...
                               mem_ctlr1/
                                        event_source_id
                                        events/
                                              partial_cacheline_read_retried/
                                              ...

So it's a bit ugly to replicate the event information across identical pmus, but that can be done via links, without too much memory cost, I assume.

Does this seem workable?

Signature · 3 lines
Regards,

- Corey
#48 lore raw
Quoted 21 lines · Corey Ashford

Just to give a concrete example, the IBM Wire-Speed Processor has four AT-"nodes" per chip, each containing four PowerPC cores.

Those four nodes together share a number of nest PMU accelerators, I/O devices, buses etc. which each have their own PMUs. Further adding to the structure is that some of the nodes are replicated. For example, we have two memory controllers, each with a pair of PMUs.

/sys/devices/system/node/node0/mem_ctlr0/

                                        event_source_id
                                        events/
                                              partial_cacheline_read_retried/
                                              partial_cacheline_write_retried/
                                              ...
                               mem_ctlr1/
                                        event_source_id
                                        events/
                                              partial_cacheline_read_retried/
                                              ...
Quoted 5 lines

So it's a bit ugly to replicate the event information across identical pmus, but that can be done via links, without too much memory cost, I assume.

Does this seem workable?

If you really have two memory controllers per node, I guess so. Sounds strange to me though, typically a memory controller is the node boundary.

But like I said in the other email, use a 1:n pmu:event_source ratio and simply stick then in the machine/device topology wherever they belong, if that ends up with multiple PMUs at one particular level, so be it.

author
#49 lore raw
In reply to Peter Zijlstra
Quoted 67 lines · Peter Zijlstra
Quoted 6 lines · Lin Ming
Quoted 3 lines

No, I'm assuming there is only 1 PMU per CPU. Corey is the expert on crazy hardware though, but I think the sanest way is to extend the CPU topology if there's more structure to it.

But our goal is to support multiple pmus, don't we need to assume there are more than 1 PMU per CPU?

No, because as I said, then its ambiguous what pmu you want. If you have that, you need to extend your topology information.

Anyway, I talked with Ingo on this and he'd like to see this somewhat extended.

Instead of a pmu_id field, which we pass into a new perf_event_attr::pmu_id field, how about creating an event_source sysfs class. Then each class can have an event_source_id and a hierarchy of 'generic' events.

We'd start using the PERF_TYPE_ space for this and express the PERF_COUNT_ space in the event attributes found inside that class.

That way we can include all the existing event enumerations into this as well.

This way we can create:

/sys/devices/system/cpu/cpuN/cpu_hardware_events

                             cpu_hardware_events/event_source_id
                             cpu_hardware_events/cpu_cycles
                             cpu_hardware_events/instructions
                                                /...

/sys/devices/system/cpu/cpuN/cpu_raw_events
cpu_raw_events/event_source_id

These would match the current PERF_TYPE_* values for compatibility

For new PMUs we can start a dynamic range of PERF_TYPE_ (say at 64k but that's not ABI and can be changed at any time, we've got u32 to play with).

For uncore this would result in:

/sys/devices/system/node/nodeN/node_raw_events
node_raw_events/event_source_id

and maybe:

/sys/devices/system/node/nodeN/node_events

                               node_events/event_source_id
                               node_events/local_misses
                                          /local_hits
                                          /remote_misses
                                          /remote_hits
                                          /...

The software events and tracepoints and kprobes stuff we could hang off of /sys/kernel/ or something

So your registration would indeed look like something:

perf_event_register_pmu(struct pmu *pmu, int type),

Quoted 8 lines

where type would normally be -1 (dynamic) but would be PERF_TYPE_ for those already laid down in ABI.

This approach will also give us a good overview
in /sys/class/event_source/, which will be a flat listing of all existing event sources.

Does this make sense?

Thanks for the idea.

Give me some time to get a clear understanding of the ideas from you and others.

And then I'll work out a patch as soon as possible.

Lin Ming

#50 lore raw
Quoted 3 lines · Lin Ming

Give me some time to get a clear understanding of the ideas from you and others.

Of course, ho hurry!

Quoted 1 line

And then I'll work out a patch as soon as possible.

Thanks for working on this!