Cavium ThunderX uncore PMU support

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

Changes to v1:
- Added NUMA support
- Fixed CPU hotplug by pmu migration
- Moved files to drivers/perf/uncore
- Removed OCX FRC and LNE drivers, these will fit better into a edac driver
- improved comments abount overflow interrupts
- removed max device limit
- trimmed include files

Feedback welcome!
Jan

-------------------------------------------------

Jan Glauber (5):
arm64/perf: Basic uncore counter support for Cavium ThunderX arm64/perf: Cavium ThunderX L2C TAD uncore support arm64/perf: Cavium ThunderX L2C CBC uncore support arm64/perf: Cavium ThunderX LMC uncore support arm64/perf: Cavium ThunderX OCX TLK uncore support

 drivers/perf/Makefile                       |   1 +
 drivers/perf/uncore/Makefile                |   5 +
 drivers/perf/uncore/uncore_cavium.c         | 314 +++++++++++++++
 drivers/perf/uncore/uncore_cavium.h         |  95 +++++
 drivers/perf/uncore/uncore_cavium_l2c_cbc.c | 237 +++++++++++
 drivers/perf/uncore/uncore_cavium_l2c_tad.c | 600 ++++++++++++++++++++++++++++
 drivers/perf/uncore/uncore_cavium_lmc.c     | 196 +++++++++
 drivers/perf/uncore/uncore_cavium_ocx_tlk.c | 380 ++++++++++++++++++
 8 files changed, 1828 insertions(+)
 create mode 100644 drivers/perf/uncore/Makefile
 create mode 100644 drivers/perf/uncore/uncore_cavium.c
 create mode 100644 drivers/perf/uncore/uncore_cavium.h
 create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_cbc.c
 create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_tad.c
 create mode 100644 drivers/perf/uncore/uncore_cavium_lmc.c
 create mode 100644 drivers/perf/uncore/uncore_cavium_ocx_tlk.c

Signature · 1 line

1.9.1

Hi Mark,

can you have a look at these patches?

Thanks,
Jan

2016-03-09 17:21 GMT+01:00 Jan Glauber <[email protected]>:

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

Changes to v1:
- Added NUMA support
- Fixed CPU hotplug by pmu migration
- Moved files to drivers/perf/uncore
- Removed OCX FRC and LNE drivers, these will fit better into a edac driver
- improved comments abount overflow interrupts
- removed max device limit
- trimmed include files

Feedback welcome!
Jan

-------------------------------------------------

Jan Glauber (5):
? arm64/perf: Basic uncore counter support for Cavium ThunderX ? arm64/perf: Cavium ThunderX L2C TAD uncore support ? arm64/perf: Cavium ThunderX L2C CBC uncore support ? arm64/perf: Cavium ThunderX LMC uncore support
? arm64/perf: Cavium ThunderX OCX TLK uncore support

?drivers/perf/Makefile? ? ? ? ? ? ? ? ? ? ? ?|? ?1 + ?drivers/perf/uncore/Makefile? ? ? ? ? ? ? ? |? ?5 + ?drivers/perf/uncore/uncore_cavium.c? ? ? ? ?| 314 +++++++++++++++ ?drivers/perf/uncore/uncore_cavium.h? ? ? ? ?|? 95 +++++ ?drivers/perf/uncore/uncore_cavium_l2c_cbc.c | 237 +++++++++++ ?drivers/perf/uncore/uncore_cavium_l2c_tad.c | 600
++++++++++++++++++++++++++++
?drivers/perf/uncore/uncore_cavium_lmc.c? ? ?| 196 +++++++++ ?drivers/perf/uncore/uncore_cavium_ocx_tlk.c | 380 ++++++++++++++++++ ?8 files changed, 1828 insertions(+)
?create mode 100644 drivers/perf/uncore/Makefile
?create mode 100644 drivers/perf/uncore/uncore_cavium.c ?create mode 100644 drivers/perf/uncore/uncore_cavium.h ?create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_cbc.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_tad.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_lmc.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_ocx_tlk.c

--
1.9.1

Provide "uncore" facilities for different non-CPU performance counter units. Based on Intel/AMD uncore pmu support.

The uncore drivers cover quite different functionality including L2 Cache, memory controllers and interconnects.

The uncore PMUs can be found under /sys/bus/event_source/devices. All counters are exported via sysfs in the corresponding events files under the PMU directory so the perf tool can list the event names.

There are some points that are special in this implementation:

1) The PMU detection relies on PCI device detection. If a matching PCI device is found the PMU is created. The code can deal with multiple units of the same type, e.g. more than one memory controller.
Note: There is also a CPUID check to determine the CPU variant, this is needed to support different hardware versions that use the same PCI IDs.

2) Counters are summarized across different units of the same type on one NUMA node.
For instance L2C TAD 0..7 are presented as a single counter (adding the values from TAD 0 to 7). Although losing the ability to read a single value the merged values are easier to use.

3) NUMA support. The device node id is used to group devices by node so counters on one node can be merged. The NUMA node can be selected via a new sysfs node attribute.
Without NUMA support all devices will be on node 0.

drivers/perf/uncore/uncore_cavium.c new file +37 -0

+void thunder_uncore_read(struct perf_event *event)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	struct thunder_uncore_unit *unit;
+	u64 prev, new = 0;
+	s64 delta;
+
+	node = get_node(hwc->config, uncore);
+
+	/*
+	 * No counter overflow interrupts so we do not
+	 * have to worry about prev_count changing on us.
+	 */
+	prev = local64_read(&hwc->prev_count);
+
+	/* read counter values from all units on the node */
+	list_for_each_entry(unit, &node->unit_list, entry)
+		new += readq(hwc->event_base + unit->map);
+
+	local64_set(&hwc->prev_count, new);
+	delta = new - prev;
+	local64_add(delta, &event->count);
+}
+
+void thunder_uncore_del(struct perf_event *event, int flags)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	int i;
+
+	event->pmu->stop(event, PERF_EF_UPDATE);
+
+	/*
+	 * For programmable counters we need to check where we installed it.

drivers/perf/uncore/uncore_cavium.c new file +14 -0

+int thunder_uncore_event_init(struct perf_event *event)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	struct thunder_uncore *uncore;
+
+	if (event->attr.type != event->pmu->type)
+		return -ENOENT;
+
+	/* we do not support sampling */
+	if (is_sampling_event(event))
+		return -EINVAL;
+
+	/* counters do not have these bits */

drivers/perf/uncore/uncore_cavium.c new file +6 -0

+		if (new_cpu >= nr_cpu_ids)
+			break;
+		perf_pmu_migrate_context(uncore->pmu, old_cpu, new_cpu);
+		cpumask_set_cpu(new_cpu, &thunder_active_mask);
+		break;
+	default:

drivers/perf/uncore/uncore_cavium.c new file +4 -0

+int __init thunder_uncore_setup(struct thunder_uncore *uncore, int device_id,
+			 unsigned long offset, unsigned long size,
+			 struct pmu *pmu, int counters)
+{

Support counters of the L2 Cache tag and data units.

Also support pass2 added/modified counters by checking MIDR.

Signed-off-by Jan Glauber <[email protected]>

---
 drivers/perf/uncore/Makefile                |   3 +-
 drivers/perf/uncore/uncore_cavium.c         |   6 +-
 drivers/perf/uncore/uncore_cavium.h         |   7 +-
 drivers/perf/uncore/uncore_cavium_l2c_tad.c | 600 ++++++++++++++++++++++++++++
 4 files changed, 613 insertions(+), 3 deletions(-)
 create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_tad.c

drivers/perf/uncore/Makefile +2 -1

index b9c72c2..6a16caf 100644
@@ -1 +1,2 @@
-obj-$(CONFIG_ARCH_THUNDER) += uncore_cavium.o
+obj-$(CONFIG_ARCH_THUNDER) += uncore_cavium.o		\
+			      uncore_cavium_l2c_tad.o

drivers/perf/uncore/uncore_cavium.c +0 -0

index 4fd5e45..b92b2ae 100644

drivers/perf/uncore/uncore_cavium.c +1 -0

 void thunder_uncore_read(struct perf_event *event)

@@ -296,6 +299,7 @@ static int __init thunder_uncore_init(void)
 		thunder_uncore_version = 1;
 	pr_info("PMU version: %d\n", thunder_uncore_version);
 
+	thunder_uncore_l2c_tad_setup();
 	return 0;
 }
 late_initcall(thunder_uncore_init);

drivers/perf/uncore/uncore_cavium.h +1 -1

index c799709..7a9c367 100644
@@ -7,7 +7,7 @@
 #define pr_fmt(fmt)     "thunderx_uncore: " fmt
 
 enum uncore_type {
-	NOP_TYPE,
+	L2C_TAD_TYPE,
 };
 
 extern int thunder_uncore_version;

drivers/perf/uncore/uncore_cavium.h +2 -0

 /* Prototypes */
 struct thunder_uncore *event_to_thunder_uncore(struct perf_event *event);
 void thunder_uncore_del(struct perf_event *event, int flags);

@@ -76,3 +79,5 @@ int thunder_uncore_setup(struct thunder_uncore *uncore, int id,
 ssize_t thunder_events_sysfs_show(struct device *dev,
 				  struct device_attribute *attr,
 				  char *page);
+
+int thunder_uncore_l2c_tad_setup(void);

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +0 -0

new file mode 100644
index 0000000..c8dc305

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +10 -0

+static void thunder_uncore_start(struct perf_event *event, int flags)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	struct thunder_uncore_unit *unit;
+	u64 prev;
+	int id;
+
+	node = get_node(hwc->config, uncore);

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +22 -0

+
+	perf_event_update_userpage(event);
+}
+
+static void thunder_uncore_stop(struct perf_event *event, int flags)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	struct thunder_uncore_unit *unit;
+
+	/* reset selection value for all units on the node */
+	node = get_node(hwc->config, uncore);
+
+	list_for_each_entry(unit, &node->unit_list, entry)
+		writeb(L2C_TAD_EVENTS_DISABLED, hwc->config_base + unit->map);
+	hwc->state |= PERF_HES_STOPPED;
+
+	if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) {
+		thunder_uncore_read(event);
+		hwc->state |= PERF_HES_UPTODATE;
+	}

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +6 -0

+
+	/* if not take the first available counter */
+	hwc->idx = -1;
+	for (i = 0; i < node->num_counters; i++) {
+		if (cmpxchg(&node->events[i], NULL, event) == NULL) {
+			hwc->idx = i;

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +17 -0

+EVENT_ATTR(l2t_hit,	L2C_TAD_EVENT_L2T_HIT);
+EVENT_ATTR(l2t_miss,	L2C_TAD_EVENT_L2T_MISS);
+EVENT_ATTR(l2t_noalloc,	L2C_TAD_EVENT_L2T_NOALLOC);
+EVENT_ATTR(l2_vic,	L2C_TAD_EVENT_L2_VIC);
+EVENT_ATTR(sc_fail,	L2C_TAD_EVENT_SC_FAIL);
+EVENT_ATTR(sc_pass,	L2C_TAD_EVENT_SC_PASS);
+EVENT_ATTR(lfb_occ,	L2C_TAD_EVENT_LFB_OCC);
+EVENT_ATTR(wait_lfb,	L2C_TAD_EVENT_WAIT_LFB);
+EVENT_ATTR(wait_vab,	L2C_TAD_EVENT_WAIT_VAB);
+EVENT_ATTR(rtg_hit,	L2C_TAD_EVENT_RTG_HIT);
+EVENT_ATTR(rtg_miss,	L2C_TAD_EVENT_RTG_MISS);
+EVENT_ATTR(l2_rtg_vic,	L2C_TAD_EVENT_L2_RTG_VIC);
+EVENT_ATTR(l2_open_oci,	L2C_TAD_EVENT_L2_OPEN_OCI);
 
+static struct attribute *thunder_l2c_tad_events_attr[] = {
+	EVENT_PTR(l2t_hit),
+	EVENT_PTR(l2t_miss),
+	EVENT_PTR(l2t_noalloc),

drivers/perf/uncore/uncore_cavium_l2c_tad.c new file +14 -0

+static struct attribute_group thunder_l2c_tad_events_group = {
+	.name = "events",
+	.attrs = NULL,
+};
+
+static const struct attribute_group *thunder_l2c_tad_attr_groups[] = {
+	&thunder_uncore_attr_group,
+	&thunder_l2c_tad_format_group,
+	&thunder_l2c_tad_events_group,
+	NULL,
+};
+
+struct pmu thunder_l2c_tad_pmu = {
+	.attr_groups	= thunder_l2c_tad_attr_groups,

drivers/perf/uncore/uncore_cavium_l2c_cbc.c new file +13 -0

+
+static void thunder_uncore_start(struct perf_event *event, int flags)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	struct thunder_uncore_unit *unit;
+	u64 prev;
+
+	node = get_node(hwc->config, uncore);
+
+	/* restore counter value divided by units into all counters */
+	if (flags & PERF_EF_RELOAD) {

drivers/perf/uncore/uncore_cavium_l2c_cbc.c new file +38 -0

+
+static int thunder_uncore_add(struct perf_event *event, int flags)
+{
+	struct thunder_uncore *uncore = event_to_thunder_uncore(event);
+	struct hw_perf_event *hwc = &event->hw;
+	struct thunder_uncore_node *node;
+	int id, i;
+
+	WARN_ON_ONCE(!uncore);
+	node = get_node(hwc->config, uncore);
+	id = get_id(hwc->config);
+
+	/* are we already assigned? */
+	if (hwc->idx != -1 && node->events[hwc->idx] == event)
+		goto out;
+
+	for (i = 0; i < node->num_counters; i++) {
+		if (node->events[i] == event) {
+			hwc->idx = i;
+			goto out;
+		}
+	}
+
+	/* these counters are self-sustained so idx must match the counter! */
+	hwc->idx = -1;
+	for (i = 0; i < node->num_counters; i++) {
+		if (l2c_cbc_events[i] == id) {
+			if (cmpxchg(&node->events[i], NULL, event) == NULL) {
+				hwc->idx = i;
+				break;
+			}
+		}
+	}
+
+out:
+	if (hwc->idx == -1)
+		return -EBUSY;
+

Mark,

are these patches still queued or should I repost them?

--Jan

Quoted 63 lines · Jan Glauber

Hi Mark,

can you have a look at these patches?

Thanks,
Jan

2016-03-09 17:21 GMT+01:00 Jan Glauber <[email protected]>:

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

Changes to v1:
- Added NUMA support
- Fixed CPU hotplug by pmu migration
- Moved files to drivers/perf/uncore
- Removed OCX FRC and LNE drivers, these will fit better into a edac driver
- improved comments abount overflow interrupts
- removed max device limit
- trimmed include files

Feedback welcome!
Jan

-------------------------------------------------

Jan Glauber (5):
? arm64/perf: Basic uncore counter support for Cavium ThunderX ? arm64/perf: Cavium ThunderX L2C TAD uncore support ? arm64/perf: Cavium ThunderX L2C CBC uncore support ? arm64/perf: Cavium ThunderX LMC uncore support
? arm64/perf: Cavium ThunderX OCX TLK uncore support

?drivers/perf/Makefile? ? ? ? ? ? ? ? ? ? ? ?|? ?1 + ?drivers/perf/uncore/Makefile? ? ? ? ? ? ? ? |? ?5 + ?drivers/perf/uncore/uncore_cavium.c? ? ? ? ?| 314 +++++++++++++++ ?drivers/perf/uncore/uncore_cavium.h? ? ? ? ?|? 95 +++++ ?drivers/perf/uncore/uncore_cavium_l2c_cbc.c | 237 +++++++++++ ?drivers/perf/uncore/uncore_cavium_l2c_tad.c | 600
++++++++++++++++++++++++++++
?drivers/perf/uncore/uncore_cavium_lmc.c? ? ?| 196 +++++++++ ?drivers/perf/uncore/uncore_cavium_ocx_tlk.c | 380 ++++++++++++++++++ ?8 files changed, 1828 insertions(+)
?create mode 100644 drivers/perf/uncore/Makefile
?create mode 100644 drivers/perf/uncore/uncore_cavium.c ?create mode 100644 drivers/perf/uncore/uncore_cavium.h ?create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_cbc.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_l2c_tad.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_lmc.c ?create mode 100644 drivers/perf/uncore/uncore_cavium_ocx_tlk.c

--
1.9.1

Quoted 20 lines · Jan Glauber

Provide "uncore" facilities for different non-CPU performance counter units. Based on Intel/AMD uncore pmu support.

The uncore drivers cover quite different functionality including L2 Cache, memory controllers and interconnects.

The uncore PMUs can be found under /sys/bus/event_source/devices. All counters are exported via sysfs in the corresponding events files under the PMU directory so the perf tool can list the event names.

There are some points that are special in this implementation:

1) The PMU detection relies on PCI device detection. If a matching PCI device is found the PMU is created. The code can deal with multiple units of the same type, e.g. more than one memory controller.
Note: There is also a CPUID check to determine the CPU variant, this is needed to support different hardware versions that use the same PCI IDs.

Quoted 5 lines

2) Counters are summarized across different units of the same type on one NUMA node.
For instance L2C TAD 0..7 are presented as a single counter (adding the values from TAD 0 to 7). Although losing the ability to read a single value the merged values are easier to use.

Merging within a NUMA node, but no further seems a little arbitrary.

Quoted 4 lines

3) NUMA support. The device node id is used to group devices by node so counters on one node can be merged. The NUMA node can be selected via a new sysfs node attribute.
Without NUMA support all devices will be on node 0.

It doesn't seem great that this depends on kernel configuration (which is independent of HW configuration). It seems confusing for the user, and fragile.

Do we not have access to another way of grouping cores (e.g. a socket ID), that's independent of kernel configuration? That seems to be how the x86 uncore PMUs are handled.

If we don't have that information, it really feels like we need additional info from FW (which would also solve the CPUID issue with point 1), or this is likely to be very fragile.

Hi Mark,

can you have a look at these patches?

Hi Jan,

Quoted 3 lines · Jan Glauber

Hi Mark,

can you have a look at these patches?

Looks like Mark reviewed this last week -- are you planning to respin?

Will

Quoted 4 lines · Will Deacon

Quoted 1 line · Jan Glauber

can you have a look at these patches?

Looks like Mark reviewed this last week -- are you planning to respin?

Yes, of course. I just had no time yet and I'm a bit lost on how to proceed without using the NUMA node information which Mark did not like to be used.

The only way to know which device is on which node would be to look at the PCI topology (which is also the source of the NUMA node_id). We could do this manually in order to not depend on CONFIG_NUMA, but I would like to know if that is acceptable before respinning the patches.

Quoted 10 lines · Jan Glauber

Quoted 4 lines · Will Deacon

Quoted 1 line · Jan Glauber

can you have a look at these patches?

Looks like Mark reviewed this last week -- are you planning to respin?

Yes, of course. I just had no time yet and I'm a bit lost on how to proceed without using the NUMA node information which Mark did not like to be used.

Quoted 5 lines

The only way to know which device is on which node would be to look at the PCI topology (which is also the source of the NUMA node_id). We could do this manually in order to not depend on CONFIG_NUMA, but I would like to know if that is acceptable before respinning the patches.

That doesn't feel like it really addresses Mark's concerns -- it's just another way to get the information that isn't a first-class PMU topology description from firmware.

Now, I don't actually mind using the NUMA topology so much in the cases where it genuinely correlates with the PMU topology. My objection is more that we end up sticking everything on node 0 if !CONFIG_NUMA, which could result in working with an incorrect PMU topology and passing all of that through to userspace.

So I'd prefer either making the driver depend on NUMA, or at the very least failing to probe the PMU if we discover a socketed system and NUMA is not selected. Do either of those work as a compromise?

Will

Quoted 27 lines · Will Deacon

Quoted 10 lines · Jan Glauber

Quoted 4 lines · Will Deacon

Quoted 1 line · Jan Glauber

can you have a look at these patches?

Looks like Mark reviewed this last week -- are you planning to respin?

Yes, of course. I just had no time yet and I'm a bit lost on how to proceed without using the NUMA node information which Mark did not like to be used.

Quoted 5 lines

The only way to know which device is on which node would be to look at the PCI topology (which is also the source of the NUMA node_id). We could do this manually in order to not depend on CONFIG_NUMA, but I would like to know if that is acceptable before respinning the patches.

That doesn't feel like it really addresses Mark's concerns -- it's just another way to get the information that isn't a first-class PMU topology description from firmware.

Now, I don't actually mind using the NUMA topology so much in the cases where it genuinely correlates with the PMU topology. My objection is more that we end up sticking everything on node 0 if !CONFIG_NUMA, which could result in working with an incorrect PMU topology and passing all of that through to userspace.

Quoted 5 lines

So I'd prefer either making the driver depend on NUMA, or at the very least failing to probe the PMU if we discover a socketed system and NUMA is not selected. Do either of those work as a compromise?

Will

That sounds like a good compromise.

So I could do the following:

1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA information to determine the device node

2) If CONFIG_NUMA is not set we check if we run on a socketed system

a) In that case we return an error and give a message that CONFIG_NUMA needs to be enabled
b) Otherwise we have a single node system and use node_id = 0

David noted that it would also be possible to extract the node id from the physical address of the device, but I'm not sure that classifies as 'first-class' topology description...

Quoted 10 lines

That sounds like a good compromise.

So I could do the following:

1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA information to determine the device node

2) If CONFIG_NUMA is not set we check if we run on a socketed system

Quoted 3 lines

a) In that case we return an error and give a message that CONFIG_NUMA needs to be enabled
b) Otherwise we have a single node system and use node_id = 0

That sounds sensible to me. How do you "check if we run on a socketed system"? My assumption would be that you could figure this out from the firmware tables?

On Tue, Apr 26, 2016 at 02:53:54PM +0100, Will Deacon wrote:

[...]

Quoted 10 lines

Quoted 10 lines

That sounds like a good compromise.

So I could do the following:

1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA information to determine the device node

2) If CONFIG_NUMA is not set we check if we run on a socketed system

Quoted 7 lines

Quoted 3 lines

a) In that case we return an error and give a message that CONFIG_NUMA needs to be enabled
b) Otherwise we have a single node system and use node_id = 0

That sounds sensible to me. How do you "check if we run on a socketed system"? My assumption would be that you could figure this out from the firmware tables?

There are probably multiple ways to detect a socketed system, with some quite hardware specific. I would like to avoid parsing DT (and ACPI) though, if possible.

A generic approach would be to do a query of the multiprocessor affinity register (MPIDR_EL1) on all CPUs. The AFF2 part (bits 23:16) contains the socket number on ThunderX. If this is non-zero on any CPU I would assume a socketed system.

Would that be feasible?

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

Hi Jan,

Quoted 6 lines · Jan Glauber

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Patches #2-5 add the various ThunderX specific PMUs.

Quoted 3 lines

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

What's the status of these patches? Were you planning to send a new version?

Will

Quoted 5 lines · Will Deacon

Quoted 4 lines · Jan Glauber

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Quoted 8 lines

Quoted 5 lines

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

What's the status of these patches? Were you planning to send a new version?

I was half-way through with addressing Mark's review comments when got side-tracked.

The principle question these patches raised remains open though in my opinion, how to determine the socket a device belongs to.

There is no first-class interface to ask a device or the firmware which socket the device lives on.

The options I see are:
A) Using NUMA node information, depends on CONFIG_NUMA
B) Decoding the socket bits of the PCI BAR address
C) Using PCI topology information

A is what I tried, but I agree that depending on CONFIG_NUMA is not a good solution. B would be easy but looks not very future-proof. So option C is what is left...

Quoted 24 lines · Jan Glauber

Quoted 5 lines · Will Deacon

Quoted 4 lines · Jan Glauber

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Quoted 8 lines

Quoted 5 lines

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

What's the status of these patches? Were you planning to send a new version?

I was half-way through with addressing Mark's review comments when got side-tracked.

The principle question these patches raised remains open though in my opinion, how to determine the socket a device belongs to.

There is no first-class interface to ask a device or the firmware which socket the device lives on.

Quoted 8 lines

The options I see are:
A) Using NUMA node information, depends on CONFIG_NUMA
B) Decoding the socket bits of the PCI BAR address
C) Using PCI topology information

A is what I tried, but I agree that depending on CONFIG_NUMA is not a good solution. B would be easy but looks not very future-proof. So option C is what is left...

Sorry to go full circle on this, but "depends on NUMA" sounds better than deriving NUMA topology from PCI to me. The only worry I have is if the NUMA information ends up being insufficient in the long-term, and we end up with a mixture of the three options above in order to figure out the PMU topology.

As long as you're happy that the PMU:NUMA topology remains 1:1, then I have no objections. The moment you need extra hacks on the side, we should probably drop the NUMA dependency altogether and figure it out some other way.

Hi Jan,

Quoted 40 lines · Will Deacon

Quoted 24 lines · Jan Glauber

Quoted 5 lines · Will Deacon

Quoted 4 lines · Jan Glauber

This patch series provides access to various counters on the ThunderX SOC.

For details of the uncore implementation see patch #1.

Quoted 8 lines

Quoted 5 lines

Patches #2-5 add the various ThunderX specific PMUs.

As suggested I've put the files under drivers/perf/uncore. I would prefer this location over drivers/bus because not all of the uncore drivers are bus related.

What's the status of these patches? Were you planning to send a new version?

I was half-way through with addressing Mark's review comments when got side-tracked.

The principle question these patches raised remains open though in my opinion, how to determine the socket a device belongs to.

There is no first-class interface to ask a device or the firmware which socket the device lives on.

Quoted 8 lines

The options I see are:
A) Using NUMA node information, depends on CONFIG_NUMA
B) Decoding the socket bits of the PCI BAR address
C) Using PCI topology information

A is what I tried, but I agree that depending on CONFIG_NUMA is not a good solution. B would be easy but looks not very future-proof. So option C is what is left...

Sorry to go full circle on this, but "depends on NUMA" sounds better than deriving NUMA topology from PCI to me. The only worry I have is if the NUMA information ends up being insufficient in the long-term, and we end up with a mixture of the three options above in order to figure out the PMU topology.

Quoted 4 lines

As long as you're happy that the PMU:NUMA topology remains 1:1, then I have no objections. The moment you need extra hacks on the side, we should probably drop the NUMA dependency altogether and figure it out some other way.

Any news on this series, or did I miss a v3? I was hoping to have this in for 4.9, but it seems to have stalled :(

Will